2022-03-01T06:20:41.921 INFO:root:teuthology version: 1.1.0-cb5b1a75 2022-03-01T06:20:41.923 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2022-02-28_21:24:59-rados-wip-yuri11-testing-2022-02-28-0803-octopus-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-02-28_21:24:59-rados-wip-yuri11-testing-2022-02-28-0803-octopus-distro-default-smithi/6712126 --owner scheduled_yuriw@teuthology --description rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} -- /home/teuthworker/archive/yuriw-2022-02-28_21:24:59-rados-wip-yuri11-testing-2022-02-28-0803-octopus-distro-default-smithi/6712126/orig.config.yaml 2022-03-01T06:20:41.952 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-01T06:20:42.038 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-02-28_21:24:59-rados-wip-yuri11-testing-2022-02-28-0803-octopus-distro-default-smithi/6712126 branch: wip-yuri11-testing-2022-02-28-0803-octopus description: rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} email: ceph-qa@ceph.io first_in_suite: false job_id: '6712126' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-02-28_21:24:59-rados-wip-yuri11-testing-2022-02-28-0803-octopus-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri11-testing-2022-02-28-0803-octopus ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 45ece157fc29b55a5ee952c81c0c23abdb89bf98 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 45ece157fc29b55a5ee952c81c0c23abdb89bf98 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri11-testing-2022-02-28-0803-octopus sha1: 45ece157fc29b55a5ee952c81c0c23abdb89bf98 owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 45ece157fc29b55a5ee952c81c0c23abdb89bf98 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri11-testing-2022-02-28-0803-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_45ece157fc29b55a5ee952c81c0c23abdb89bf98/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 45ece157fc29b55a5ee952c81c0c23abdb89bf98 targets: smithi169.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD8SUEToNAAZDStj/SDDr/QlgEIorjZKX+qog40d8RN2STgmz4czBNaDimIJ++6vGglCO8RuRnbOnCmlGlHmj4m5xaDubxLn5wXQQIeMLv705+C3kl1jRcwacP2y2Vyd4z6BF4C/WNY0BeKttksP89NZSMtvulaei3TjudwKeofWqpT5j/DyU9W4TkBvV9Wv4d0ZMqfU5lIXfRZqeY0UCqj5awoXl64InRMsDyC7mKw3AsaYcuvr+vYAGAiexnQ8Q6fSj9SKYRipN6eRcaeEatAMPvIxwPQ0cEo5+yZ4NfzeDfj5/isbwNJS8PJ5grqirhkFntvsmjXGJc+24QXMPem/ID41T+3usfDDYMiQK5ZUbEiRZpMcjllhwXeB4u+oMdtDEiubA27m1nYR9SBO+MsBoWdJn0uOgSxKlKI0VCY/Msa+D29G6L8eodvvr2/OjxZwHcXIsg6X+MOxwHubV5PYOLJdq0UfjluFxemc5lbENTMIpHRaotCWM2yxP54q50= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master teuthology_sha1: cb5b1a75970b7aa81700a0ab081943b3146d40d0 timestamp: 2022-02-28_21:24:59 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-01T06:20:42.038 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_45ece157fc29b55a5ee952c81c0c23abdb89bf98/qa; will attempt to use it 2022-03-01T06:20:42.039 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_45ece157fc29b55a5ee952c81c0c23abdb89bf98/qa/tasks 2022-03-01T06:20:42.039 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-01T06:20:42.041 INFO:teuthology.task.internal:Checking packages... 2022-03-01T06:20:42.065 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '45ece157fc29b55a5ee952c81c0c23abdb89bf98' 2022-03-01T06:20:42.066 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-01T06:20:42.067 INFO:teuthology.packaging:ref: None 2022-03-01T06:20:42.068 INFO:teuthology.packaging:tag: None 2022-03-01T06:20:42.068 INFO:teuthology.packaging:branch: wip-yuri11-testing-2022-02-28-0803-octopus 2022-03-01T06:20:42.069 INFO:teuthology.packaging:sha1: 45ece157fc29b55a5ee952c81c0c23abdb89bf98 2022-03-01T06:20:42.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri11-testing-2022-02-28-0803-octopus 2022-03-01T06:20:42.297 INFO:teuthology.task.internal:Found packages for ceph version 15.2.15-249.g45ece157 2022-03-01T06:20:42.298 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-01T06:20:42.315 INFO:teuthology.task.internal:no buildpackages task found 2022-03-01T06:20:42.315 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-01T06:20:42.418 INFO:teuthology.task.internal:Saving configuration 2022-03-01T06:20:42.429 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-01T06:20:42.492 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-01T06:20:42.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-28_21:24:59-rados-wip-yuri11-testing-2022-02-28-0803-octopus-distro-default-smithi/6712126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-01 06:15:33.757854', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGr4yFPu9NsCQQGQcDmD49sUMoa2zByGxhUEplEH4KS6LflgtyUvEF2Fgw+7NIlfXT6kGKuhl286Zp8GK6sqfk='} 2022-03-01T06:20:42.516 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-01T06:20:42.599 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2022-03-01T06:20:42.600 INFO:teuthology.run_tasks:Running task console_log... 2022-03-01T06:20:42.897 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff89e08bd90>, signals=[15]) 2022-03-01T06:20:42.898 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-01T06:20:43.046 INFO:teuthology.task.internal:Opening connections... 2022-03-01T06:20:43.047 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2022-03-01T06:20:43.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-01T06:20:43.132 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-01T06:20:43.336 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2022-03-01T06:20:43.364 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2022-03-01T06:20:43.365 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2022-03-01T06:20:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2022-03-01T06:20:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="8" 2022-03-01T06:20:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2022-03-01T06:20:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2022-03-01T06:20:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="8" 2022-03-01T06:20:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el8" 2022-03-01T06:20:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-01T06:20:43.430 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2022-03-01T06:20:43.430 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-01T06:20:43.430 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2022-03-01T06:20:43.430 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-01T06:20:43.430 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-01T06:20:43.431 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-01T06:20:43.432 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2022-03-01T06:20:43.460 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-01T06:20:43.572 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-01T06:20:43.847 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-01T06:20:43.850 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2022-03-01T06:20:43.869 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-01T06:20:44.106 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-01T06:20:44.106 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2022-03-01T06:20:44.143 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-01T06:20:44.320 INFO:teuthology.run_tasks:Running task kernel... 2022-03-01T06:20:44.344 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-01T06:20:44.345 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-03-01T06:20:44.345 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2022-03-01T06:20:44.366 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-358.el8.x86_64 2022-03-01T06:20:44.366 INFO:teuthology.task.kernel:Running kernel on smithi169: 4.18.0-358.el8.x86_64 2022-03-01T06:20:44.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2022-03-01T06:20:45.358 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:20:45.358 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:20:47.213 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - AppStream 32 MB/s | 20 MB 00:00 2022-03-01T06:20:51.192 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - Base 36 MB/s | 19 MB 00:00 2022-03-01T06:20:54.071 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - Extras 154 kB/s | 18 kB 00:00 2022-03-01T06:20:54.431 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - PowerTools 17 MB/s | 4.2 MB 00:00 2022-03-01T06:20:57.906 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2022-03-01T06:20:59.164 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 11 MB 00:01 2022-03-01T06:21:01.872 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 429 kB/s | 24 kB 00:00 2022-03-01T06:21:04.327 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-358.el8.x86_64 is already installed. 2022-03-01T06:21:04.328 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-03-01T06:21:04.422 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:21:04.423 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:21:04.424 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2022-03-01T06:21:04.424 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:21:04.425 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:21:04.425 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 4.18.0-365.el8 CentOS-Base 7.7 M 2022-03-01T06:21:04.425 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 4.18.0-365.el8 CentOS-Base 39 M 2022-03-01T06:21:04.425 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 4.18.0-365.el8 CentOS-Base 31 M 2022-03-01T06:21:04.426 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:21:04.426 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:21:04.426 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:21:04.427 INFO:teuthology.orchestra.run.smithi169.stdout:Install 3 Packages 2022-03-01T06:21:04.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:21:04.427 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 78 M 2022-03-01T06:21:04.427 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 92 M 2022-03-01T06:21:04.428 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:21:05.082 INFO:teuthology.orchestra.run.smithi169.stdout:(1/3): kernel-4.18.0-365.el8.x86_64.rpm 12 MB/s | 7.7 MB 00:00 2022-03-01T06:21:06.040 INFO:teuthology.orchestra.run.smithi169.stdout:(2/3): kernel-modules-4.18.0-365.el8.x86_64.rpm 19 MB/s | 31 MB 00:01 2022-03-01T06:21:06.490 INFO:teuthology.orchestra.run.smithi169.stdout:(3/3): kernel-core-4.18.0-365.el8.x86_64.rpm 19 MB/s | 39 MB 00:02 2022-03-01T06:21:06.494 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:21:06.494 INFO:teuthology.orchestra.run.smithi169.stdout:Total 38 MB/s | 78 MB 00:02 2022-03-01T06:21:07.074 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:21:07.442 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:21:07.442 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:21:08.793 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:21:08.812 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:21:11.392 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:21:11.881 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-4.18.0-365.el8.x86_64 1/3 2022-03-01T06:21:15.508 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-4.18.0-365.el8.x86_64 1/3 2022-03-01T06:21:15.894 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-4.18.0-365.el8.x86_64 2/3 2022-03-01T06:21:19.054 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-4.18.0-365.el8.x86_64 2/3 2022-03-01T06:21:19.307 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-4.18.0-365.el8.x86_64 3/3 2022-03-01T06:21:46.964 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-4.18.0-365.el8.x86_64 3/3 2022-03-01T06:21:47.313 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-4.18.0-365.el8.x86_64 3/3 2022-03-01T06:21:47.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-4.18.0-365.el8.x86_64 1/3 2022-03-01T06:21:47.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-4.18.0-365.el8.x86_64 2/3 2022-03-01T06:21:47.798 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-4.18.0-365.el8.x86_64 3/3 2022-03-01T06:21:47.799 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:21:47.799 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:21:47.799 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-4.18.0-365.el8.x86_64 kernel-core-4.18.0-365.el8.x86_64 2022-03-01T06:21:47.799 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-4.18.0-365.el8.x86_64 2022-03-01T06:21:47.800 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:21:47.800 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:21:47.890 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel --last | head -n 1 2022-03-01T06:21:48.077 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-365.el8.x86_64 Tue 01 Mar 2022 06:21:19 AM UTC 2022-03-01T06:21:48.079 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-365.el8.x86_64 2022-03-01T06:21:48.079 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-358.el8.x86_64 Expected: 4.18.0-365.el8.x86_64 2022-03-01T06:21:48.080 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-01T06:21:48.080 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-01T06:21:48.080 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2022-03-01T06:21:48.080 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2022-03-01T06:21:48.081 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel --last | head -n 1 2022-03-01T06:21:48.250 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-365.el8.x86_64 Tue 01 Mar 2022 06:21:19 AM UTC 2022-03-01T06:21:48.251 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-365.el8.x86_64 2022-03-01T06:21:48.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2022-03-01T06:21:48.285 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2022-03-01T06:21:48.285 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2022-03-01T06:21:48.286 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.02 2022-03-01T06:21:48.286 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 106.el8 2022-03-01T06:21:48.286 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2022-03-01T06:21:48.286 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 04 Feb 2022 04:06:37 PM UTC 2022-03-01T06:21:48.287 INFO:teuthology.orchestra.run.smithi169.stdout:Group : System Environment/Base 2022-03-01T06:21:48.287 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 9265339 2022-03-01T06:21:48.287 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2022-03-01T06:21:48.287 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 08 Nov 2021 06:49:52 AM UTC, Key ID 05b555b38483c65d 2022-03-01T06:21:48.288 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.02-106.el8.src.rpm 2022-03-01T06:21:48.288 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Mon 08 Nov 2021 06:39:57 AM UTC 2022-03-01T06:21:48.288 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : kbuilder.bsys.centos.org 2022-03-01T06:21:48.288 INFO:teuthology.orchestra.run.smithi169.stdout:Relocations : (not relocatable) 2022-03-01T06:21:48.288 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : CentOS BuildSystem 2022-03-01T06:21:48.289 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2022-03-01T06:21:48.289 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2022-03-01T06:21:48.289 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2022-03-01T06:21:48.289 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2022-03-01T06:21:48.290 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:21:48.290 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-03-01T06:21:48.290 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2022-03-01T06:21:48.290 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2022-03-01T06:21:48.291 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2022-03-01T06:21:48.291 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:21:48.291 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2022-03-01T06:21:48.292 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-03-01T06:21:48.292 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 4.18.0-365.el8.x86_64 2022-03-01T06:21:48.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-03-01T06:21:49.604 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2022-03-01T06:21:51.273 INFO:teuthology.orchestra.run.smithi169.stderr:done 2022-03-01T06:21:51.275 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2022-03-01T06:21:51.306 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.BskUNSOtkL 2022-03-01T06:21:51.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BskUNSOtkL 2022-03-01T06:21:51.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.BskUNSOtkL 2022-03-01T06:21:51.534 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.BskUNSOtkL is 6KB 2022-03-01T06:21:51.584 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.BskUNSOtkL 2022-03-01T06:21:51.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2022-03-01T06:21:51.666 INFO:teuthology.orchestra.run.smithi169.stdout:de9f16d36b2643dd85552c363843f45b-0-rescue.conf 2022-03-01T06:21:51.667 INFO:teuthology.orchestra.run.smithi169.stdout:de9f16d36b2643dd85552c363843f45b-4.18.0-305.3.1.el8.x86_64.conf 2022-03-01T06:21:51.667 INFO:teuthology.orchestra.run.smithi169.stdout:de9f16d36b2643dd85552c363843f45b-4.18.0-358.el8.x86_64.conf 2022-03-01T06:21:51.668 INFO:teuthology.orchestra.run.smithi169.stdout:de9f16d36b2643dd85552c363843f45b-4.18.0-365.el8.x86_64.conf 2022-03-01T06:21:51.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default de9f16d36b2643dd85552c363843f45b-4.18.0-365.el8.x86_64 2022-03-01T06:21:51.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2022-03-01T06:22:21.878 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2022-03-01T06:22:21.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-03-01T06:22:21.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-01T06:22:40.341 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2022-03-01T06:23:10.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-03-01T06:23:10.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-01T06:23:13.451 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2022-03-01T06:23:43.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-03-01T06:23:43.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-01T06:23:44.049 DEBUG:teuthology.orchestra.run.smithi169:> true 2022-03-01T06:23:44.341 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-03-01T06:23:44.341 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-365.el8.x86_64"... 2022-03-01T06:23:44.342 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2022-03-01T06:23:44.357 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-365.el8.x86_64 2022-03-01T06:23:44.358 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-365.el8.x86_64 vs 4.18.0-365.el8.x86_64 2022-03-01T06:23:44.358 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-01T06:23:44.359 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2022-03-01T06:23:45.360 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-03-01T06:23:45.436 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-01T06:23:45.836 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2022-03-01T06:23:45.883 DEBUG:teuthology.parallel:result is None 2022-03-01T06:23:45.883 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-01T06:23:45.892 INFO:teuthology.task.internal:Creating test directory... 2022-03-01T06:23:45.893 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-01T06:23:46.290 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-01T06:23:46.307 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-01T06:23:46.327 INFO:teuthology.task.internal:Creating archive directory... 2022-03-01T06:23:46.328 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-01T06:23:46.789 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-01T06:23:46.799 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-01T06:23:46.800 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-01T06:23:46.900 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:23:46.913 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:23:46.914 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-01T06:23:46.926 INFO:teuthology.task.internal:Configuring sudo... 2022-03-01T06:23:46.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-01T06:23:46.954 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-01T06:23:46.967 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-01T06:23:46.968 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-01T06:23:47.010 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-01T06:23:47.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-01T06:23:47.212 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-01T06:23:47.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-01T06:23:47.297 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2022-03-01T06:23:47.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-01T06:23:47.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2022-03-01T06:23:47.676 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-03-01T06:23:47.695 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-01T06:23:47.708 INFO:teuthology.task.internal:Starting timer... 2022-03-01T06:23:47.708 INFO:teuthology.run_tasks:Running task pcp... 2022-03-01T06:23:47.745 INFO:teuthology.run_tasks:Running task selinux... 2022-03-01T06:23:47.779 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-03-01T06:23:47.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2022-03-01T06:23:47.848 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2022-03-01T06:23:47.850 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-01T06:23:47.850 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2022-03-01T06:23:47.950 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2022-03-01T06:23:47.951 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2022-03-01T06:23:47.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-03-01T06:23:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-01T06:23:47.978 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-01T06:23:47.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2022-03-01T06:23:48.054 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-01T06:23:48.069 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-01T06:23:48.070 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-01T06:23:48.097 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-01T06:23:48.099 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-01T06:28:23.511 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2022-03-01T06:28:23.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-03-01T06:28:23.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-01T06:28:23.585 DEBUG:teuthology.orchestra.run.smithi169:> true 2022-03-01T06:28:23.656 INFO:teuthology.run_tasks:Running task clock... 2022-03-01T06:28:23.675 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-01T06:28:23.677 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-01T06:28:23.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-01T06:28:23.729 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-03-01T06:28:23.743 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-03-01T06:28:23.812 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2022-03-01T06:28:23.824 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2022-03-01T06:28:23.838 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-03-01T06:28:23.852 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-03-01T06:28:23.918 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2022-03-01T06:28:23.922 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-03-01T06:28:23.922 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2022-03-01T06:28:23.923 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-01T06:28:23.923 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-01T06:28:23.923 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-01T06:28:23.924 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-03-01T06:28:23.924 INFO:teuthology.run_tasks:Running task install... 2022-03-01T06:28:23.933 DEBUG:teuthology.task.install:project ceph 2022-03-01T06:28:23.934 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '45ece157fc29b55a5ee952c81c0c23abdb89bf98'}} 2022-03-01T06:28:23.934 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '45ece157fc29b55a5ee952c81c0c23abdb89bf98'} 2022-03-01T06:28:23.935 INFO:teuthology.task.install:Using flavor: default 2022-03-01T06:28:23.942 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-01T06:28:23.942 INFO:teuthology.task.install:extra packages: [] 2022-03-01T06:28:23.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo lsb_release -is 2022-03-01T06:28:23.998 INFO:teuthology.orchestra.run.smithi169.stdout:CentOSStream 2022-03-01T06:28:23.999 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-03-01T06:28:24.000 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '45ece157fc29b55a5ee952c81c0c23abdb89bf98', 'tag': None, 'wait_for_package': False} 2022-03-01T06:28:24.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo lsb_release -is 2022-03-01T06:28:24.074 INFO:teuthology.orchestra.run.smithi169.stdout:CentOSStream 2022-03-01T06:28:24.076 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-03-01T06:28:24.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=45ece157fc29b55a5ee952c81c0c23abdb89bf98 2022-03-01T06:28:24.297 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2022-02-28-0803-octopus/45ece157fc29b55a5ee952c81c0c23abdb89bf98/centos/8/flavors/default/ 2022-03-01T06:28:24.297 INFO:teuthology.task.install.rpm:Package version is 15.2.15-249.g45ece157 2022-03-01T06:28:24.474 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2022-02-28-0803-octopus/45ece157fc29b55a5ee952c81c0c23abdb89bf98/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2022-02-28-0803-octopus/45ece157fc29b55a5ee952c81c0c23abdb89bf98/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2022-02-28-0803-octopus/45ece157fc29b55a5ee952c81c0c23abdb89bf98/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-03-01T06:28:24.474 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2022-03-01T06:28:24.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-03-01T06:28:24.501 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-03-01T06:28:24.501 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/45ece157fc29b55a5ee952c81c0c23abdb89bf98/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-03-01T06:28:24.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-03-01T06:28:24.645 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-03-01T06:28:24.711 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2022-03-01T06:28:24.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2022-03-01T06:28:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:28:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:28:24.942 INFO:teuthology.orchestra.run.smithi169.stdout:56 files removed 2022-03-01T06:28:24.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2022-03-01T06:28:25.139 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:28:25.139 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:28:25.693 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 183 kB/s | 69 kB 00:00 2022-03-01T06:28:26.270 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 28 kB/s | 15 kB 00:00 2022-03-01T06:28:26.620 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 4.4 kB/s | 1.4 kB 00:00 2022-03-01T06:28:27.189 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - AppStream 36 MB/s | 20 MB 00:00 2022-03-01T06:28:31.182 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - Base 37 MB/s | 19 MB 00:00 2022-03-01T06:28:34.098 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - Extras 156 kB/s | 18 kB 00:00 2022-03-01T06:28:34.441 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-8 - PowerTools 18 MB/s | 4.2 MB 00:00 2022-03-01T06:28:38.015 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2022-03-01T06:28:39.217 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 10 MB/s | 11 MB 00:01 2022-03-01T06:28:41.940 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 441 kB/s | 24 kB 00:00 2022-03-01T06:28:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:28:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:28:44.488 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:28:44.489 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:28:44.489 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:28:44.489 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:15.2.15-249.g45ece157.el8 ceph 10 M 2022-03-01T06:28:44.489 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:28:44.490 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:15.2.15-249.g45ece157.el8 ceph 5.6 M 2022-03-01T06:28:44.490 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:15.2.15-249.g45ece157.el8 ceph 21 M 2022-03-01T06:28:44.490 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:15.2.15-249.g45ece157.el8 ceph 26 k 2022-03-01T06:28:44.490 INFO:teuthology.orchestra.run.smithi169.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2022-03-01T06:28:44.491 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-03-01T06:28:44.491 INFO:teuthology.orchestra.run.smithi169.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2022-03-01T06:28:44.491 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:15.2.15-249.g45ece157.el8 ceph 509 k 2022-03-01T06:28:44.491 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-03-01T06:28:44.492 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-03-01T06:28:44.492 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:15.2.15-249.g45ece157.el8 ceph 3.5 M 2022-03-01T06:28:44.492 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:15.2.15-249.g45ece157.el8 ceph 380 k 2022-03-01T06:28:44.492 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:15.2.15-249.g45ece157.el8 ceph 1.9 M 2022-03-01T06:28:44.493 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-03-01T06:28:44.493 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:15.2.15-249.g45ece157.el8 ceph 3.2 M 2022-03-01T06:28:44.493 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-03-01T06:28:44.494 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-03-01T06:28:44.494 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse 2022-03-01T06:28:44.494 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 2:15.2.15-249.g45ece157.el8 ceph 40 k 2022-03-01T06:28:44.494 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:15.2.15-249.g45ece157.el8 ceph 77 k 2022-03-01T06:28:44.495 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:15.2.15-249.g45ece157.el8 ceph 164 k 2022-03-01T06:28:44.495 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-03-01T06:28:44.495 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:15.2.15-249.g45ece157.el8 ceph 364 k 2022-03-01T06:28:44.495 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:15.2.15-249.g45ece157.el8 ceph 349 k 2022-03-01T06:28:44.496 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:15.2.15-249.g45ece157.el8 ceph 113 k 2022-03-01T06:28:44.496 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:28:44.496 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:28:44.496 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:28:44.497 INFO:teuthology.orchestra.run.smithi169.stdout:Install 24 Packages 2022-03-01T06:28:44.497 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:28:44.498 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2022-03-01T06:28:44.498 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 162 M 2022-03-01T06:28:44.498 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:28:46.041 INFO:teuthology.orchestra.run.smithi169.stdout:(1/24): ceph-base-15.2.15-249.g45ece157.el8.x86 3.6 MB/s | 5.6 MB 00:01 2022-03-01T06:28:46.150 INFO:teuthology.orchestra.run.smithi169.stdout:(2/24): ceph-selinux-15.2.15-249.g45ece157.el8. 236 kB/s | 26 kB 00:00 2022-03-01T06:28:46.517 INFO:teuthology.orchestra.run.smithi169.stdout:(3/24): libcephfs2-15.2.15-249.g45ece157.el8.x8 1.4 MB/s | 509 kB 00:00 2022-03-01T06:28:46.951 INFO:teuthology.orchestra.run.smithi169.stdout:(4/24): ceph-radosgw-15.2.15-249.g45ece157.el8. 4.1 MB/s | 10 MB 00:02 2022-03-01T06:28:47.059 INFO:teuthology.orchestra.run.smithi169.stdout:(5/24): libradosstriper1-15.2.15-249.g45ece157. 3.4 MB/s | 380 kB 00:00 2022-03-01T06:28:47.376 INFO:teuthology.orchestra.run.smithi169.stdout:(6/24): librbd1-15.2.15-249.g45ece157.el8.x86_6 6.1 MB/s | 1.9 MB 00:00 2022-03-01T06:28:47.885 INFO:teuthology.orchestra.run.smithi169.stdout:(7/24): librgw2-15.2.15-249.g45ece157.el8.x86_6 6.3 MB/s | 3.2 MB 00:00 2022-03-01T06:28:47.986 INFO:teuthology.orchestra.run.smithi169.stdout:(8/24): python3-ceph-argparse-15.2.15-249.g45ec 395 kB/s | 40 kB 00:00 2022-03-01T06:28:48.085 INFO:teuthology.orchestra.run.smithi169.stdout:(9/24): python3-ceph-common-15.2.15-249.g45ece1 775 kB/s | 77 kB 00:00 2022-03-01T06:28:48.169 INFO:teuthology.orchestra.run.smithi169.stdout:(10/24): python3-cephfs-15.2.15-249.g45ece157.e 1.9 MB/s | 164 kB 00:00 2022-03-01T06:28:48.261 INFO:teuthology.orchestra.run.smithi169.stdout:(11/24): python3-rados-15.2.15-249.g45ece157.el 3.9 MB/s | 364 kB 00:00 2022-03-01T06:28:48.403 INFO:teuthology.orchestra.run.smithi169.stdout:(12/24): librados2-15.2.15-249.g45ece157.el8.x8 1.9 MB/s | 3.5 MB 00:01 2022-03-01T06:28:48.436 INFO:teuthology.orchestra.run.smithi169.stdout:(13/24): python3-rbd-15.2.15-249.g45ece157.el8. 1.9 MB/s | 349 kB 00:00 2022-03-01T06:28:48.495 INFO:teuthology.orchestra.run.smithi169.stdout:(14/24): python3-rgw-15.2.15-249.g45ece157.el8. 1.2 MB/s | 113 kB 00:00 2022-03-01T06:28:48.562 INFO:teuthology.orchestra.run.smithi169.stdout:(15/24): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2022-03-01T06:28:48.587 INFO:teuthology.orchestra.run.smithi169.stdout:(16/24): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2022-03-01T06:28:48.647 INFO:teuthology.orchestra.run.smithi169.stdout:(17/24): librabbitmq-0.9.0-3.el8.x86_64.rpm 792 kB/s | 47 kB 00:00 2022-03-01T06:28:48.679 INFO:teuthology.orchestra.run.smithi169.stdout:(18/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2022-03-01T06:28:48.922 INFO:teuthology.orchestra.run.smithi169.stdout:(19/24): fmt-6.2.1-1.el8.x86_64.rpm 424 kB/s | 116 kB 00:00 2022-03-01T06:28:49.014 INFO:teuthology.orchestra.run.smithi169.stdout:(20/24): gperftools-libs-2.7-9.el8.x86_64.rpm 918 kB/s | 306 kB 00:00 2022-03-01T06:28:49.055 INFO:teuthology.orchestra.run.smithi169.stdout:(21/24): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2022-03-01T06:28:49.081 INFO:teuthology.orchestra.run.smithi169.stdout:(22/24): liboath-2.6.2-3.el8.x86_64.rpm 879 kB/s | 59 kB 00:00 2022-03-01T06:28:49.131 INFO:teuthology.orchestra.run.smithi169.stdout:(23/24): libunwind-1.3.1-3.el8.x86_64.rpm 998 kB/s | 75 kB 00:00 2022-03-01T06:28:49.364 INFO:teuthology.orchestra.run.smithi169.stdout:(24/24): ceph-common-15.2.15-249.g45ece157.el8. 4.3 MB/s | 21 MB 00:04 2022-03-01T06:28:49.368 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:28:49.369 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.9 MB/s | 48 MB 00:04 2022-03-01T06:28:49.412 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:28:49.456 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:28:49.457 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:28:49.940 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:28:49.943 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:28:50.929 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:28:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/24 2022-03-01T06:28:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/24 2022-03-01T06:28:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/24 2022-03-01T06:28:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/24 2022-03-01T06:28:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados2-2:15.2.15-249.g45ece157.el8.x86_64 4/24 2022-03-01T06:28:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:15.2.15-249.g45ece157.el8.x86_64 4/24 2022-03-01T06:28:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:15.2.15-249.g45ece157.el8.x86_64 5/24 2022-03-01T06:28:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:15.2.15-249.g45ece157.el8.x86_64 6/24 2022-03-01T06:28:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:15.2.15-249.g45ece157.el8.x86_64 6/24 2022-03-01T06:28:52.964 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librbd1-2:15.2.15-249.g45ece157.el8.x86_64 7/24 2022-03-01T06:28:53.031 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:15.2.15-249.g45ece157.el8.x86_64 7/24 2022-03-01T06:28:53.255 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/24 2022-03-01T06:28:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/24 2022-03-01T06:28:53.734 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/24 2022-03-01T06:28:53.897 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:15.2.15-249.g45ece157.el8.x86_64 10/24 2022-03-01T06:28:53.976 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:15.2.15-249.g45ece157.el8.x86_64 10/24 2022-03-01T06:28:54.127 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/24 2022-03-01T06:28:54.290 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:15.2.15-249.g45ece157.el8. 12/24 2022-03-01T06:28:54.458 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:15.2.15-249.g45ece157.el8.x86_64 13/24 2022-03-01T06:28:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:15.2.15-249.g45ece157.el8.x86_64 14/24 2022-03-01T06:28:54.896 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:15.2.15-249.g45ece157.el8.x86_64 15/24 2022-03-01T06:28:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:15.2.15-249.g45ece157.el8.x86_6 16/24 2022-03-01T06:28:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:15.2.15-249.g45ece157.el8.x86_6 16/24 2022-03-01T06:28:55.307 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/24 2022-03-01T06:28:55.464 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/24 2022-03-01T06:28:55.633 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/24 2022-03-01T06:28:55.831 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:15.2.15-249.g45ece157.el8.x8 20/24 2022-03-01T06:28:59.141 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:15.2.15-249.g45ece157.el8.x86_64 21/24 2022-03-01T06:28:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:15.2.15-249.g45ece157.el8.x86_64 21/24 2022-03-01T06:29:00.004 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:15.2.15-249.g45ece157.el8.x86_64 21/24 2022-03-01T06:29:00.259 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:15.2.15-249.g45ece157.el8.x86_64 22/24 2022-03-01T06:29:00.348 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:15.2.15-249.g45ece157.el8.x86_64 22/24 2022-03-01T06:29:00.521 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:15.2.15-249.g45ece157.el8.x86_64 23/24 2022-03-01T06:29:17.676 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:15.2.15-249.g45ece157.el8.x86_64 23/24 2022-03-01T06:29:17.677 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2022-03-01T06:29:17.677 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2022-03-01T06:29:17.677 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2022-03-01T06:29:17.677 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /run 2022-03-01T06:29:17.678 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2022-03-01T06:29:17.678 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2022-03-01T06:29:17.678 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2022-03-01T06:29:17.679 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2022-03-01T06:29:17.679 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2022-03-01T06:29:17.679 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:18.706 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:15.2.15-249.g45ece157.el8.x86_64 24/24 2022-03-01T06:29:19.219 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:15.2.15-249.g45ece157.el8.x86_64 24/24 2022-03-01T06:29:19.219 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:15.2.15-249.g45ece157.el8.x86_64 1/24 2022-03-01T06:29:19.219 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:15.2.15-249.g45ece157.el8.x86_64 2/24 2022-03-01T06:29:19.220 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:15.2.15-249.g45ece157.el8.x86_64 3/24 2022-03-01T06:29:19.220 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:15.2.15-249.g45ece157.el8.x86_64 4/24 2022-03-01T06:29:19.220 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:15.2.15-249.g45ece157.el8.x86_64 5/24 2022-03-01T06:29:19.220 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:15.2.15-249.g45ece157.el8.x86_64 6/24 2022-03-01T06:29:19.221 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:15.2.15-249.g45ece157.el8.x86_6 7/24 2022-03-01T06:29:19.221 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:15.2.15-249.g45ece157.el8.x86_64 8/24 2022-03-01T06:29:19.221 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:15.2.15-249.g45ece157.el8.x86_64 9/24 2022-03-01T06:29:19.221 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:15.2.15-249.g45ece157.el8. 10/24 2022-03-01T06:29:19.222 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:15.2.15-249.g45ece157.el8.x8 11/24 2022-03-01T06:29:19.222 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:15.2.15-249.g45ece157.el8.x86_64 12/24 2022-03-01T06:29:19.222 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:15.2.15-249.g45ece157.el8.x86_64 13/24 2022-03-01T06:29:19.222 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:15.2.15-249.g45ece157.el8.x86_64 14/24 2022-03-01T06:29:19.223 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:15.2.15-249.g45ece157.el8.x86_64 15/24 2022-03-01T06:29:19.223 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 16/24 2022-03-01T06:29:19.223 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 17/24 2022-03-01T06:29:19.224 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/24 2022-03-01T06:29:19.224 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/24 2022-03-01T06:29:19.224 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 20/24 2022-03-01T06:29:19.224 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 21/24 2022-03-01T06:29:19.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : leveldb-1.22-1.el8.x86_64 22/24 2022-03-01T06:29:19.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 23/24 2022-03-01T06:29:20.200 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 24/24 2022-03-01T06:29:20.200 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:20.201 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:29:20.201 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.201 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.202 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.202 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.202 INFO:teuthology.orchestra.run.smithi169.stdout: fmt-6.2.1-1.el8.x86_64 2022-03-01T06:29:20.203 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-03-01T06:29:20.203 INFO:teuthology.orchestra.run.smithi169.stdout: leveldb-1.22-1.el8.x86_64 2022-03-01T06:29:20.203 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.204 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.2-3.el8.x86_64 2022-03-01T06:29:20.204 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-03-01T06:29:20.204 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.204 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.205 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.205 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-03-01T06:29:20.206 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.206 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.3.1-3.el8.x86_64 2022-03-01T06:29:20.206 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-03-01T06:29:20.207 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.207 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.207 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.208 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-03-01T06:29:20.208 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.209 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.209 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:20.209 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:20.210 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:20.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2022-03-01T06:29:20.626 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:20.626 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:21.019 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:40 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:21.815 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:21.815 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:21.816 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:29:21.816 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:21.816 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:29:21.816 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:15.2.15-249.g45ece157.el8 ceph 42 M 2022-03-01T06:29:21.817 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:29:21.817 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-03-01T06:29:21.817 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-03-01T06:29:21.817 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-03-01T06:29:21.818 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-03-01T06:29:21.818 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:21.818 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:29:21.818 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:21.819 INFO:teuthology.orchestra.run.smithi169.stdout:Install 5 Packages 2022-03-01T06:29:21.819 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:21.819 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 43 M 2022-03-01T06:29:21.820 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 154 M 2022-03-01T06:29:21.820 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:29:21.885 INFO:teuthology.orchestra.run.smithi169.stdout:(1/5): jq-1.6-3.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2022-03-01T06:29:21.919 INFO:teuthology.orchestra.run.smithi169.stdout:(2/5): socat-1.7.4.1-1.el8.x86_64.rpm 11 MB/s | 323 kB 00:00 2022-03-01T06:29:21.944 INFO:teuthology.orchestra.run.smithi169.stdout:(3/5): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2022-03-01T06:29:22.118 INFO:teuthology.orchestra.run.smithi169.stdout:(4/5): xmlstarlet-1.6.1-11.el8.x86_64.rpm 347 kB/s | 69 kB 00:00 2022-03-01T06:29:24.136 INFO:teuthology.orchestra.run.smithi169.stdout:(5/5): ceph-test-15.2.15-249.g45ece157.el8.x86_ 18 MB/s | 42 MB 00:02 2022-03-01T06:29:24.139 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:29:24.140 INFO:teuthology.orchestra.run.smithi169.stdout:Total 18 MB/s | 43 MB 00:02 2022-03-01T06:29:24.173 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:29:24.192 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:29:24.193 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:29:24.454 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:29:24.457 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:29:24.885 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:29:25.070 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/5 2022-03-01T06:29:25.307 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2022-03-01T06:29:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/5 2022-03-01T06:29:25.515 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/5 2022-03-01T06:29:29.352 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-3.el8.x86_64 4/5 2022-03-01T06:29:29.640 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:15.2.15-249.g45ece157.el8.x86_64 5/5 2022-03-01T06:29:29.906 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:15.2.15-249.g45ece157.el8.x86_64 5/5 2022-03-01T06:29:29.907 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:15.2.15-249.g45ece157.el8.x86_64 1/5 2022-03-01T06:29:29.907 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-3.el8.x86_64 2/5 2022-03-01T06:29:29.907 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/5 2022-03-01T06:29:29.908 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2022-03-01T06:29:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 5/5 2022-03-01T06:29:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:30.320 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:29:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:15.2.15-249.g45ece157.el8.x86_64 jq-1.6-3.el8.x86_64 2022-03-01T06:29:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2022-03-01T06:29:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-03-01T06:29:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:30.321 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:30.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2022-03-01T06:29:30.674 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:30.675 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:31.076 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:50 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:31.875 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:31.878 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2022-03-01T06:29:31.879 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:29:31.879 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2022-03-01T06:29:31.879 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:29:31.879 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:15.2.15-249.g45ece157.el8 ceph 6.4 k 2022-03-01T06:29:31.880 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:29:31.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:15.2.15-249.g45ece157.el8 ceph 2.0 M 2022-03-01T06:29:31.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:15.2.15-249.g45ece157.el8 ceph 1.4 M 2022-03-01T06:29:31.880 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 192 k 2022-03-01T06:29:31.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:15.2.15-249.g45ece157.el8 ceph 4.4 M 2022-03-01T06:29:31.881 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:15.2.15-249.g45ece157.el8 ceph 17 M 2022-03-01T06:29:31.881 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-03-01T06:29:31.881 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-03-01T06:29:31.881 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-03-01T06:29:31.882 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-03-01T06:29:31.882 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-03-01T06:29:31.882 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-03-01T06:29:31.882 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-03-01T06:29:31.883 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-03-01T06:29:31.883 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-03-01T06:29:31.883 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-03-01T06:29:31.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-03-01T06:29:31.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-03-01T06:29:31.884 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-03-01T06:29:31.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-03-01T06:29:31.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-03-01T06:29:31.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-03-01T06:29:31.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-03-01T06:29:31.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-03-01T06:29:31.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-03-01T06:29:31.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-03-01T06:29:31.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-03-01T06:29:31.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-03-01T06:29:31.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-03-01T06:29:31.888 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-03-01T06:29:31.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-03-01T06:29:31.889 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-03-01T06:29:31.889 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:31.889 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:29:31.890 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2022-03-01T06:29:31.890 INFO:teuthology.orchestra.run.smithi169.stdout:Install 32 Packages 2022-03-01T06:29:31.890 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:31.891 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 28 M 2022-03-01T06:29:31.891 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 91 M 2022-03-01T06:29:31.892 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:29:32.144 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-15.2.15-249.g45ece157.el8.x86_64.r 25 kB/s | 6.4 kB 00:00 2022-03-01T06:29:32.353 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-mgr-15.2.15-249.g45ece157.el8.x86_ 3.1 MB/s | 1.4 MB 00:00 2022-03-01T06:29:32.403 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): ceph-mds-15.2.15-249.g45ece157.el8.x86_ 3.9 MB/s | 2.0 MB 00:00 2022-03-01T06:29:32.495 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): ceph-mgr-modules-core-15.2.15-249.g45ec 2.0 MB/s | 192 kB 00:00 2022-03-01T06:29:32.553 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-03-01T06:29:32.571 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 2.3 MB/s | 39 kB 00:00 2022-03-01T06:29:32.604 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.1 MB/s | 103 kB 00:00 2022-03-01T06:29:32.621 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2022-03-01T06:29:32.663 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 11 MB/s | 457 kB 00:00 2022-03-01T06:29:32.688 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2022-03-01T06:29:32.731 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2022-03-01T06:29:32.756 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-03-01T06:29:32.923 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2022-03-01T06:29:33.065 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2022-03-01T06:29:33.273 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): ceph-osd-15.2.15-249.g45ece157.el8.x86 18 MB/s | 17 MB 00:00 2022-03-01T06:29:33.299 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 739 kB/s | 173 kB 00:00 2022-03-01T06:29:33.349 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2022-03-01T06:29:33.399 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2022-03-01T06:29:33.441 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2022-03-01T06:29:33.508 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): ceph-mon-15.2.15-249.g45ece157.el8.x86 3.2 MB/s | 4.4 MB 00:01 2022-03-01T06:29:33.533 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 639 kB/s | 59 kB 00:00 2022-03-01T06:29:33.583 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2022-03-01T06:29:33.609 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2022-03-01T06:29:33.634 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2022-03-01T06:29:33.660 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2022-03-01T06:29:33.718 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2022-03-01T06:29:33.743 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 329 kB/s | 27 kB 00:00 2022-03-01T06:29:33.769 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2022-03-01T06:29:33.827 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2022-03-01T06:29:33.852 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.8 MB/s | 241 kB 00:00 2022-03-01T06:29:33.877 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2022-03-01T06:29:33.911 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2022-03-01T06:29:33.914 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:29:33.914 INFO:teuthology.orchestra.run.smithi169.stdout:Total 14 MB/s | 28 MB 00:02 2022-03-01T06:29:33.992 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:29:34.029 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:29:34.029 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:29:34.309 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:29:34.312 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:29:35.463 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:29:35.695 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-03-01T06:29:35.906 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-03-01T06:29:36.128 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-03-01T06:29:36.330 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-03-01T06:29:36.511 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-03-01T06:29:36.657 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-03-01T06:29:36.859 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-03-01T06:29:36.995 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-03-01T06:29:37.195 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-03-01T06:29:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-03-01T06:29:37.683 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-03-01T06:29:37.918 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-03-01T06:29:38.372 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-03-01T06:29:38.601 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-03-01T06:29:38.823 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-03-01T06:29:39.031 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-03-01T06:29:39.224 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-03-01T06:29:39.518 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-03-01T06:29:39.729 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-03-01T06:29:39.892 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-03-01T06:29:40.103 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-03-01T06:29:40.377 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-03-01T06:29:40.661 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-03-01T06:29:40.889 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-03-01T06:29:41.209 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-03-01T06:29:41.517 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-03-01T06:29:41.906 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:15.2.15-249.g45ece157.el8. 27/32 2022-03-01T06:29:42.110 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:15.2.15-249.g45ece157.el8.x86_64 28/32 2022-03-01T06:29:43.485 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:15.2.15-249.g45ece157.el8.x86_64 28/32 2022-03-01T06:29:43.727 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:15.2.15-249.g45ece157.el8.x86_64 29/32 2022-03-01T06:29:44.098 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:15.2.15-249.g45ece157.el8.x86_64 29/32 2022-03-01T06:29:44.297 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:15.2.15-249.g45ece157.el8.x86_64 30/32 2022-03-01T06:29:44.480 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:15.2.15-249.g45ece157.el8.x86_64 30/32 2022-03-01T06:29:44.664 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:15.2.15-249.g45ece157.el8.x86_64 31/32 2022-03-01T06:29:44.680 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:15.2.15-249.g45ece157.el8.x86_64 31/32 2022-03-01T06:29:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:15.2.15-249.g45ece157.el8.x86_64 32/32 2022-03-01T06:29:45.230 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:15.2.15-249.g45ece157.el8.x86_64 32/32 2022-03-01T06:29:45.230 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:15.2.15-249.g45ece157.el8.x86_64 1/32 2022-03-01T06:29:45.231 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:15.2.15-249.g45ece157.el8.x86_64 2/32 2022-03-01T06:29:45.231 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:15.2.15-249.g45ece157.el8.x86_64 3/32 2022-03-01T06:29:45.231 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:15.2.15-249.g45ece157.el8.x86_64 4/32 2022-03-01T06:29:45.231 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:15.2.15-249.g45ece157.el8.x86_64 5/32 2022-03-01T06:29:45.232 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:15.2.15-249.g45ece157.el8. 6/32 2022-03-01T06:29:45.232 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-03-01T06:29:45.232 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-03-01T06:29:45.233 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-03-01T06:29:45.233 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-03-01T06:29:45.233 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-03-01T06:29:45.233 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-03-01T06:29:45.233 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-03-01T06:29:45.234 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-03-01T06:29:45.234 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-03-01T06:29:45.234 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-03-01T06:29:45.234 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-03-01T06:29:45.235 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-03-01T06:29:45.235 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-03-01T06:29:45.235 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-03-01T06:29:45.235 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-03-01T06:29:45.236 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-03-01T06:29:45.236 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-03-01T06:29:45.236 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-03-01T06:29:45.236 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-03-01T06:29:45.237 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-03-01T06:29:45.237 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-03-01T06:29:45.238 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-03-01T06:29:45.238 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-03-01T06:29:45.239 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-03-01T06:29:45.239 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-03-01T06:29:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-03-01T06:29:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:46.271 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:29:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:29:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-03-01T06:29:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-03-01T06:29:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-03-01T06:29:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-03-01T06:29:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-03-01T06:29:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-03-01T06:29:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-6.2-6.el8.noarch 2022-03-01T06:29:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-03-01T06:29:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-03-01T06:29:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.0.6-13.el8.noarch 2022-03-01T06:29:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-03-01T06:29:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-03-01T06:29:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-03-01T06:29:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-2.6-1.el8.noarch 2022-03-01T06:29:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-03-01T06:29:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.14-14.el8.noarch 2022-03-01T06:29:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2017.2-9.el8.noarch 2022-03-01T06:29:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-03-01T06:29:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-03-01T06:29:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-03-01T06:29:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-03-01T06:29:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-03-01T06:29:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-03-01T06:29:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-03-01T06:29:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-03-01T06:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-03-01T06:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:46.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2022-03-01T06:29:46.625 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:46.626 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:47.021 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:06 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:47.764 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:29:47.812 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:47.813 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:29:47.814 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:47.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2022-03-01T06:29:48.044 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:48.045 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:48.444 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:07 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:49.238 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:49.239 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:49.240 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:29:49.240 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:49.240 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:29:49.240 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 59 k 2022-03-01T06:29:49.241 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:49.241 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:29:49.241 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:49.241 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:29:49.242 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:49.242 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 59 k 2022-03-01T06:29:49.242 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 223 k 2022-03-01T06:29:49.242 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:29:49.582 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-15.2.15-249.g45ece157.el8.noarch.rpm 171 kB/s | 59 kB 00:00 2022-03-01T06:29:49.583 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:29:49.584 INFO:teuthology.orchestra.run.smithi169.stdout:Total 171 kB/s | 59 kB 00:00 2022-03-01T06:29:49.584 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:29:49.590 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:29:49.591 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:29:49.592 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:29:49.595 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:29:49.652 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:29:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:15.2.15-249.g45ece157.el8.noarch 1/1 2022-03-01T06:29:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:15.2.15-249.g45ece157.el8.noarch 1/1 2022-03-01T06:29:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:15.2.15-249.g45ece157.el8.noarch 1/1 2022-03-01T06:29:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:15.2.15-249.g45ece157.el8.noarch 1/1 2022-03-01T06:29:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:51.900 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:29:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:29:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:51.901 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:52.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2022-03-01T06:29:52.431 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:52.431 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:52.933 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:11 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:53.771 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:53.773 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:53.773 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2022-03-01T06:29:53.773 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:53.773 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:29:53.774 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:15.2.15-249.g45ece157.el8 ceph 203 k 2022-03-01T06:29:53.774 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:53.774 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:29:53.774 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:53.775 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:29:53.775 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:53.775 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 203 k 2022-03-01T06:29:53.775 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 603 k 2022-03-01T06:29:53.776 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:29:54.121 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-15.2.15-249.g45ece1 585 kB/s | 203 kB 00:00 2022-03-01T06:29:54.122 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:29:54.122 INFO:teuthology.orchestra.run.smithi169.stdout:Total 583 kB/s | 203 kB 00:00 2022-03-01T06:29:54.123 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:29:54.133 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:29:54.133 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:29:54.179 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:29:54.182 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:29:54.292 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:29:54.454 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:15.2.15-249.g45ece157. 1/1 2022-03-01T06:29:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.2.15-249.g45ece157. 1/1 2022-03-01T06:29:55.168 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:15.2.15-249.g45ece157. 1/1 2022-03-01T06:29:55.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:55.168 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:29:55.169 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:29:55.169 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:55.170 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:55.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2022-03-01T06:29:55.549 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:55.549 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:55.952 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:14 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:56.692 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:29:56.739 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:56.740 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:29:56.740 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:29:56.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2022-03-01T06:29:56.982 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:56.983 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:29:57.379 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:16 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:29:58.173 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:29:58.174 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:58.175 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:29:58.175 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:58.175 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:29:58.176 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 3.6 M 2022-03-01T06:29:58.176 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:29:58.176 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 24 k 2022-03-01T06:29:58.177 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 9.0 k 2022-03-01T06:29:58.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-03-01T06:29:58.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-03-01T06:29:58.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-03-01T06:29:58.178 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:58.178 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:29:58.178 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:29:58.178 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2022-03-01T06:29:58.179 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:29:58.179 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.9 M 2022-03-01T06:29:58.180 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 19 M 2022-03-01T06:29:58.180 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:29:58.749 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): ceph-prometheus-alerts-15.2.15-249.g45ec 16 kB/s | 9.0 kB 00:00 2022-03-01T06:29:59.119 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): ceph-grafana-dashboards-15.2.15-249.g45e 26 kB/s | 24 kB 00:00 2022-03-01T06:29:59.479 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 58 kB/s | 43 kB 00:00 2022-03-01T06:29:59.584 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 72 kB/s | 34 kB 00:00 2022-03-01T06:29:59.641 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2022-03-01T06:30:00.158 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-mgr-dashboard-15.2.15-249.g45ece157 1.8 MB/s | 3.6 MB 00:01 2022-03-01T06:30:00.161 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:00.162 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.0 MB/s | 3.9 MB 00:01 2022-03-01T06:30:00.173 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:00.184 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:00.184 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:00.344 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:00.347 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:00.634 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:00.867 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-03-01T06:30:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-03-01T06:30:01.280 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-03-01T06:30:01.425 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:15.2.15-249.g45ece157.el8.n 4/6 2022-03-01T06:30:02.470 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:15.2.15-249.g45ece157.el8. 5/6 2022-03-01T06:30:02.732 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:15.2.15-249.g45ece157.el8.noarc 6/6 2022-03-01T06:30:03.004 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.2.15-249.g45ece157.el8.noarc 6/6 2022-03-01T06:30:03.005 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:15.2.15-249.g45ece157.el8. 1/6 2022-03-01T06:30:03.005 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:15.2.15-249.g45ece157.el8.noarc 2/6 2022-03-01T06:30:03.005 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:15.2.15-249.g45ece157.el8.n 3/6 2022-03-01T06:30:03.005 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-03-01T06:30:03.006 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-03-01T06:30:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-03-01T06:30:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:03.392 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:30:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:30:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:30:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-03-01T06:30:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-03-01T06:30:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.4.1-12.el8.noarch 2022-03-01T06:30:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:03.394 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:03.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2022-03-01T06:30:03.665 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:03.666 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:04.061 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:23 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:04.845 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:04.846 INFO:teuthology.orchestra.run.smithi169.stdout:===================================================================================================== 2022-03-01T06:30:04.847 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:30:04.847 INFO:teuthology.orchestra.run.smithi169.stdout:===================================================================================================== 2022-03-01T06:30:04.847 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:04.847 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 8.6 M 2022-03-01T06:30:04.848 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:30:04.848 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 8.5.0-10.el8 CentOS-Base 643 k 2022-03-01T06:30:04.848 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 8.5.0-10.el8 CentOS-Base 171 k 2022-03-01T06:30:04.848 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-03-01T06:30:04.849 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-03-01T06:30:04.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-03-01T06:30:04.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-03-01T06:30:04.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-03-01T06:30:04.849 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:04.850 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:04.850 INFO:teuthology.orchestra.run.smithi169.stdout:===================================================================================================== 2022-03-01T06:30:04.851 INFO:teuthology.orchestra.run.smithi169.stdout:Install 8 Packages 2022-03-01T06:30:04.851 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:04.852 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2022-03-01T06:30:04.852 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 210 M 2022-03-01T06:30:04.852 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:05.513 INFO:teuthology.orchestra.run.smithi169.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 7.1 MB/s | 4.7 MB 00:00 2022-03-01T06:30:05.747 INFO:teuthology.orchestra.run.smithi169.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2022-03-01T06:30:05.822 INFO:teuthology.orchestra.run.smithi169.stdout:(3/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 5.0 MB/s | 4.9 MB 00:00 2022-03-01T06:30:05.847 INFO:teuthology.orchestra.run.smithi169.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2022-03-01T06:30:05.897 INFO:teuthology.orchestra.run.smithi169.stdout:(5/8): libgfortran-8.5.0-10.el8.x86_64.rpm 13 MB/s | 643 kB 00:00 2022-03-01T06:30:05.923 INFO:teuthology.orchestra.run.smithi169.stdout:(6/8): libquadmath-8.5.0-10.el8.x86_64.rpm 6.7 MB/s | 171 kB 00:00 2022-03-01T06:30:06.348 INFO:teuthology.orchestra.run.smithi169.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2022-03-01T06:30:06.498 INFO:teuthology.orchestra.run.smithi169.stdout:(8/8): ceph-mgr-diskprediction-local-15.2.15-24 5.2 MB/s | 8.6 MB 00:01 2022-03-01T06:30:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Total 22 MB/s | 37 MB 00:01 2022-03-01T06:30:06.728 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:06.746 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:06.747 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:07.070 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:07.073 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:07.617 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:07.768 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 1/8 2022-03-01T06:30:07.912 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 1/8 2022-03-01T06:30:08.110 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 2/8 2022-03-01T06:30:08.706 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 2/8 2022-03-01T06:30:08.868 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-03-01T06:30:09.483 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-03-01T06:30:09.626 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-03-01T06:30:10.307 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-03-01T06:30:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-03-01T06:30:10.876 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-03-01T06:30:12.640 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-03-01T06:30:13.970 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-03-01T06:30:14.184 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:15.2.15-249.g45ece15 8/8 2022-03-01T06:30:14.472 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.2.15-249.g45ece15 8/8 2022-03-01T06:30:14.472 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.2.15-249.g45ece15 1/8 2022-03-01T06:30:14.472 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-03-01T06:30:14.473 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-03-01T06:30:14.473 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-03-01T06:30:14.473 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-03-01T06:30:14.473 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-03-01T06:30:14.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 7/8 2022-03-01T06:30:14.883 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 8/8 2022-03-01T06:30:14.883 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:14.883 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:14.884 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:30:14.884 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-03-01T06:30:14.884 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-03-01T06:30:14.884 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.15-3.el8.x86_64 2022-03-01T06:30:14.885 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-03-01T06:30:14.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-03-01T06:30:14.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-03-01T06:30:14.885 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-03-01T06:30:14.886 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:14.887 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:14.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2022-03-01T06:30:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:15.161 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:15.561 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:34 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:16.354 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:16.355 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2022-03-01T06:30:16.356 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:30:16.356 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2022-03-01T06:30:16.356 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:16.356 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 27 k 2022-03-01T06:30:16.357 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:30:16.357 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-03-01T06:30:16.358 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-03-01T06:30:16.358 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-03-01T06:30:16.358 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-03-01T06:30:16.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-03-01T06:30:16.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-03-01T06:30:16.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-03-01T06:30:16.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-03-01T06:30:16.360 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-03-01T06:30:16.360 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-03-01T06:30:16.360 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-03-01T06:30:16.361 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-03-01T06:30:16.361 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:16.361 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:16.361 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2022-03-01T06:30:16.362 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2022-03-01T06:30:16.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:16.363 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.9 M 2022-03-01T06:30:16.363 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 24 M 2022-03-01T06:30:16.364 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:16.440 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2022-03-01T06:30:16.465 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 189 kB/s | 20 kB 00:00 2022-03-01T06:30:16.499 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.3 MB/s | 110 kB 00:00 2022-03-01T06:30:16.524 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.1 MB/s | 126 kB 00:00 2022-03-01T06:30:16.549 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2022-03-01T06:30:16.575 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2022-03-01T06:30:16.617 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): ceph-mgr-rook-15.2.15-249.g45ece157.el8 106 kB/s | 27 kB 00:00 2022-03-01T06:30:16.658 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 303 kB/s | 33 kB 00:00 2022-03-01T06:30:16.700 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2022-03-01T06:30:16.767 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 540 kB/s | 81 kB 00:00 2022-03-01T06:30:16.784 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 738 kB/s | 61 kB 00:00 2022-03-01T06:30:16.818 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-03-01T06:30:16.868 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.6 MB/s | 1.2 MB 00:00 2022-03-01T06:30:16.869 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:16.870 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.7 MB/s | 1.9 MB 00:00 2022-03-01T06:30:16.921 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:16.936 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:16.937 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:17.015 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:17.018 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:17.490 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:17.735 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-03-01T06:30:17.925 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-03-01T06:30:18.152 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-03-01T06:30:18.356 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-03-01T06:30:18.555 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-03-01T06:30:18.811 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-03-01T06:30:19.136 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-03-01T06:30:19.329 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-03-01T06:30:20.373 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-03-01T06:30:20.831 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-03-01T06:30:21.040 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-03-01T06:30:21.260 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-03-01T06:30:21.449 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:15.2.15-249.g45ece157.el8.noarch 13/13 2022-03-01T06:30:21.729 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:15.2.15-249.g45ece157.el8.noarch 13/13 2022-03-01T06:30:21.729 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:15.2.15-249.g45ece157.el8.noarch 1/13 2022-03-01T06:30:21.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-03-01T06:30:21.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-03-01T06:30:21.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-03-01T06:30:21.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-03-01T06:30:21.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-03-01T06:30:21.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-03-01T06:30:21.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-03-01T06:30:21.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-03-01T06:30:21.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-03-01T06:30:21.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-03-01T06:30:21.733 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-03-01T06:30:22.169 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-03-01T06:30:22.169 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:22.170 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:22.170 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:30:22.170 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-03-01T06:30:22.171 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-03-01T06:30:22.171 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-03-01T06:30:22.171 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-03-01T06:30:22.171 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-03-01T06:30:22.172 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-03-01T06:30:22.172 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-03-01T06:30:22.172 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-03-01T06:30:22.173 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-03-01T06:30:22.173 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-03-01T06:30:22.173 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.8-1.el8.noarch 2022-03-01T06:30:22.174 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-03-01T06:30:22.174 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:22.174 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:22.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2022-03-01T06:30:22.443 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:22.444 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:22.841 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:41 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:23.627 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:23.627 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:23.628 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:30:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:23.629 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:15.2.15-249.g45ece157.el8 ceph-noarch 68 k 2022-03-01T06:30:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:30:23.629 INFO:teuthology.orchestra.run.smithi169.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2022-03-01T06:30:23.629 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-03-01T06:30:23.630 INFO:teuthology.orchestra.run.smithi169.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2022-03-01T06:30:23.630 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-03-01T06:30:23.630 INFO:teuthology.orchestra.run.smithi169.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2022-03-01T06:30:23.630 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2022-03-01T06:30:23.632 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.8 M 2022-03-01T06:30:23.633 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 24 M 2022-03-01T06:30:23.633 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:23.736 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.0 MB/s | 538 kB 00:00 2022-03-01T06:30:23.869 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2022-03-01T06:30:23.886 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 141 kB/s | 21 kB 00:00 2022-03-01T06:30:23.906 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): ceph-mgr-cephadm-15.2.15-249.g45ece157.e 247 kB/s | 68 kB 00:00 2022-03-01T06:30:23.990 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 471 kB/s | 49 kB 00:00 2022-03-01T06:30:24.074 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.7 MB/s | 355 kB 00:00 2022-03-01T06:30:24.075 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:24.076 INFO:teuthology.orchestra.run.smithi169.stdout:Total 13 MB/s | 5.8 MB 00:00 2022-03-01T06:30:24.126 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:24.135 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:24.135 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:24.229 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:24.232 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:24.530 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:24.780 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2022-03-01T06:30:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2022-03-01T06:30:26.000 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2022-03-01T06:30:26.493 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2022-03-01T06:30:26.715 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2022-03-01T06:30:26.931 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:15.2.15-249.g45ece157.el8.noarch 6/6 2022-03-01T06:30:27.184 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.2.15-249.g45ece157.el8.noarch 6/6 2022-03-01T06:30:27.185 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:15.2.15-249.g45ece157.el8.noarch 1/6 2022-03-01T06:30:27.185 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2022-03-01T06:30:27.185 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2022-03-01T06:30:27.186 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2022-03-01T06:30:27.186 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2022-03-01T06:30:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2022-03-01T06:30:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:27.565 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:15.2.15-249.g45ece157.el8.noarch 2022-03-01T06:30:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: python3-apipkg-1.5-6.el8.noarch 2022-03-01T06:30:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.5.1-7.el8.noarch 2022-03-01T06:30:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: python3-execnet-1.7.1-1.el8.noarch 2022-03-01T06:30:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-03-01T06:30:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: python3-remoto-1.2.1-1.el8.noarch 2022-03-01T06:30:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:27.567 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:27.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2022-03-01T06:30:27.823 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:28.229 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:47 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:29.055 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:29.056 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2022-03-01T06:30:29.056 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:29.056 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:15.2.15-249.g45ece157.el8 ceph 574 k 2022-03-01T06:30:29.057 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2022-03-01T06:30:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2022-03-01T06:30:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:29.057 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:29.058 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:29.058 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2022-03-01T06:30:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:29.059 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 657 k 2022-03-01T06:30:29.059 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.8 M 2022-03-01T06:30:29.059 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:29.124 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.2 MB/s | 82 kB 00:00 2022-03-01T06:30:29.424 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-15.2.15-249.g45ece157.el8.x86_ 1.5 MB/s | 574 kB 00:00 2022-03-01T06:30:29.425 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:29.427 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.7 MB/s | 657 kB 00:00 2022-03-01T06:30:29.436 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:29.456 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:29.457 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:29.535 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:29.539 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:29.953 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2022-03-01T06:30:30.147 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:15.2.15-249.g45ece157.el8.x86_64 2/2 2022-03-01T06:30:30.503 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:15.2.15-249.g45ece157.el8.x86_64 2/2 2022-03-01T06:30:30.504 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:15.2.15-249.g45ece157.el8.x86_64 1/2 2022-03-01T06:30:30.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2022-03-01T06:30:30.888 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:30.888 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:30.888 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:15.2.15-249.g45ece157.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2022-03-01T06:30:30.888 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:30.889 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:31.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2022-03-01T06:30:31.302 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:31.302 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:31.707 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:50 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:32.500 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:32.501 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:32.501 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2022-03-01T06:30:32.502 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:32.502 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:32.502 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:15.2.15-249.g45ece157.el8 ceph 140 k 2022-03-01T06:30:32.503 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:32.503 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:32.503 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:32.503 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:30:32.503 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:32.504 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 140 k 2022-03-01T06:30:32.504 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 478 k 2022-03-01T06:30:32.504 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:32.769 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-15.2.15-249.g45ece157.el8.x86_64 525 kB/s | 140 kB 00:00 2022-03-01T06:30:32.770 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:32.771 INFO:teuthology.orchestra.run.smithi169.stdout:Total 523 kB/s | 140 kB 00:00 2022-03-01T06:30:32.771 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:32.775 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:32.776 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:32.824 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:32.828 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:32.933 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:33.112 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:33.388 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:33.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:33.790 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:33.790 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:33.790 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:30:33.790 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:33.791 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:33.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2022-03-01T06:30:34.151 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:34.152 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:34.559 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:53 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:35.341 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:35.389 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:35.390 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:35.391 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:35.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2022-03-01T06:30:35.631 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:35.632 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:36.037 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:55 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:36.829 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:36.830 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:36.830 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2022-03-01T06:30:36.831 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:36.831 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:36.831 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:15.2.15-249.g45ece157.el8 ceph 22 k 2022-03-01T06:30:36.831 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:36.832 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:36.832 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:36.832 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:30:36.832 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:36.833 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 22 k 2022-03-01T06:30:36.833 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 78 k 2022-03-01T06:30:36.833 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:37.071 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-15.2.15-249.g45ece157.el8.x86_6 91 kB/s | 22 kB 00:00 2022-03-01T06:30:37.072 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:37.072 INFO:teuthology.orchestra.run.smithi169.stdout:Total 90 kB/s | 22 kB 00:00 2022-03-01T06:30:37.072 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:37.075 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:37.075 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:37.077 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:37.080 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:37.134 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:37.295 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:37.528 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:37.863 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:37.863 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:37.864 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:37.864 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:30:37.864 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:37.864 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:37.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2022-03-01T06:30:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:38.517 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:57 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:39.267 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:39.316 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:39.317 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:39.318 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:39.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2022-03-01T06:30:39.567 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:39.567 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:39.964 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:58 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:40.713 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:40.762 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:40.763 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:40.763 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:40.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2022-03-01T06:30:41.009 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:41.009 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:41.409 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:00 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:42.153 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:42.201 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:42.203 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:42.203 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:42.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2022-03-01T06:30:42.569 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:42.569 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:42.973 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:01 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:43.724 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:43.773 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:43.774 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:43.774 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:43.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2022-03-01T06:30:44.156 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:44.157 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:44.556 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:03 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:45.337 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:45.385 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:45.386 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:45.387 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:45.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2022-03-01T06:30:45.630 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:45.630 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:46.031 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:05 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:46.785 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:15.2.15-249.g45ece157.el8.x86_64 is already installed. 2022-03-01T06:30:46.833 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:46.834 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2022-03-01T06:30:46.834 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:46.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2022-03-01T06:30:47.072 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:47.073 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:47.471 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:06 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:48.260 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:48.261 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:48.261 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2022-03-01T06:30:48.261 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:48.262 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:48.262 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:15.2.15-249.g45ece157.el8 ceph 72 k 2022-03-01T06:30:48.262 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:48.262 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:48.263 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:48.263 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:30:48.263 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:48.263 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 72 k 2022-03-01T06:30:48.264 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 184 k 2022-03-01T06:30:48.264 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:48.655 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-15.2.15-249.g45ece157.el8.x86_64.rpm 184 kB/s | 72 kB 00:00 2022-03-01T06:30:48.661 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:48.662 INFO:teuthology.orchestra.run.smithi169.stdout:Total 184 kB/s | 72 kB 00:00 2022-03-01T06:30:48.662 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:48.665 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:48.665 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:48.714 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:48.717 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:48.841 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:49.273 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:49.633 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:49.633 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:49.633 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:49.633 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:30:49.634 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:49.634 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:49.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2022-03-01T06:30:50.002 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:50.002 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:50.412 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:09 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:51.210 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:51.211 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2022-03-01T06:30:51.212 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:51.212 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:15.2.15-249.g45ece157.el8 ceph 2.7 M 2022-03-01T06:30:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:51.213 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:51.213 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:51.214 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:30:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:51.214 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.7 M 2022-03-01T06:30:51.214 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 9.8 M 2022-03-01T06:30:51.214 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:52.349 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-15.2.15-249.g45ece157.el8.x86_64.rpm 2.4 MB/s | 2.7 MB 00:01 2022-03-01T06:30:52.350 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:52.350 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.4 MB/s | 2.7 MB 00:01 2022-03-01T06:30:52.351 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:52.362 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:52.362 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:52.424 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:52.427 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:53.019 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:53.420 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:53.799 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:53.799 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:53.799 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:53.800 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:30:53.800 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:53.800 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:53.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2022-03-01T06:30:54.165 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:54.166 INFO:teuthology.orchestra.run.smithi169.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-03-01T06:30:54.571 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:13 ago on Tue 01 Mar 2022 06:28:41 AM UTC. 2022-03-01T06:30:55.403 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2022-03-01T06:30:55.404 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:55.404 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2022-03-01T06:30:55.405 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:55.405 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2022-03-01T06:30:55.405 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:15.2.15-249.g45ece157.el8 ceph 140 k 2022-03-01T06:30:55.405 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:55.406 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2022-03-01T06:30:55.406 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2022-03-01T06:30:55.406 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2022-03-01T06:30:55.407 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:55.407 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 140 k 2022-03-01T06:30:55.407 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 418 k 2022-03-01T06:30:55.407 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2022-03-01T06:30:55.704 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-15.2.15-249.g45ece157.el8.x86_64.rpm 469 kB/s | 140 kB 00:00 2022-03-01T06:30:55.705 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2022-03-01T06:30:55.706 INFO:teuthology.orchestra.run.smithi169.stdout:Total 467 kB/s | 140 kB 00:00 2022-03-01T06:30:55.706 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2022-03-01T06:30:55.715 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2022-03-01T06:30:55.715 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2022-03-01T06:30:55.763 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2022-03-01T06:30:55.766 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2022-03-01T06:30:55.879 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2022-03-01T06:30:56.121 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:56.747 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:15.2.15-249.g45ece157.el8.x86_64 1/1 2022-03-01T06:30:56.747 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:56.748 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2022-03-01T06:30:56.748 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:15.2.15-249.g45ece157.el8.x86_64 2022-03-01T06:30:56.748 INFO:teuthology.orchestra.run.smithi169.stdout: 2022-03-01T06:30:56.748 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2022-03-01T06:30:56.921 DEBUG:teuthology.parallel:result is None 2022-03-01T06:30:56.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo lsb_release -is 2022-03-01T06:30:56.957 INFO:teuthology.orchestra.run.smithi169.stdout:CentOSStream 2022-03-01T06:30:56.959 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-03-01T06:30:56.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=45ece157fc29b55a5ee952c81c0c23abdb89bf98 2022-03-01T06:30:57.185 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-03-01T06:30:57.206 INFO:teuthology.orchestra.run.smithi169.stdout:15.2.15-249.g45ece157.el8 2022-03-01T06:30:57.207 INFO:teuthology.packaging:The installed version of ceph is 15.2.15-249.g45ece157.el8 2022-03-01T06:30:57.208 INFO:teuthology.task.install:The correct ceph version 15.2.15-249.g45ece157 is installed. 2022-03-01T06:30:57.208 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-01T06:30:57.209 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2022-03-01T06:30:57.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-01T06:30:57.273 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-01T06:30:57.273 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2022-03-01T06:30:57.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2022-03-01T06:30:57.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-01T06:30:57.405 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-01T06:30:57.405 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2022-03-01T06:30:57.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-01T06:30:57.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-01T06:30:57.538 INFO:teuthology.run_tasks:Running task workunit... 2022-03-01T06:30:57.553 INFO:tasks.workunit:Pulling workunits from ref 45ece157fc29b55a5ee952c81c0c23abdb89bf98 2022-03-01T06:30:57.555 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-03-01T06:30:57.555 INFO:tasks.workunit:timeout=3h 2022-03-01T06:30:57.555 INFO:tasks.workunit:cleanup=True 2022-03-01T06:30:57.556 DEBUG:teuthology.orchestra.run.smithi169:> stat -- /home/ubuntu/cephtest/mnt.0 2022-03-01T06:30:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-01T06:30:57.595 INFO:teuthology.orchestra.run.smithi169.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-03-01T06:30:57.595 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -- /home/ubuntu/cephtest/mnt.0 2022-03-01T06:30:57.652 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2022-03-01T06:30:57.652 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2022-03-01T06:30:57.710 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 45ece157fc29b55a5ee952c81c0c23abdb89bf98 2022-03-01T06:30:57.766 INFO:tasks.workunit.client.0.smithi169.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-03-01T06:31:34.614 INFO:tasks.workunit.client.0.smithi169.stderr:Note: switching to '45ece157fc29b55a5ee952c81c0c23abdb89bf98'. 2022-03-01T06:31:34.615 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.615 INFO:tasks.workunit.client.0.smithi169.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-01T06:31:34.615 INFO:tasks.workunit.client.0.smithi169.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-01T06:31:34.616 INFO:tasks.workunit.client.0.smithi169.stderr:state without impacting any branches by switching back to a branch. 2022-03-01T06:31:34.616 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.616 INFO:tasks.workunit.client.0.smithi169.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-01T06:31:34.616 INFO:tasks.workunit.client.0.smithi169.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-01T06:31:34.617 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.617 INFO:tasks.workunit.client.0.smithi169.stderr: git switch -c 2022-03-01T06:31:34.617 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.617 INFO:tasks.workunit.client.0.smithi169.stderr:Or undo this operation with: 2022-03-01T06:31:34.618 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.618 INFO:tasks.workunit.client.0.smithi169.stderr: git switch - 2022-03-01T06:31:34.618 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.618 INFO:tasks.workunit.client.0.smithi169.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-01T06:31:34.618 INFO:tasks.workunit.client.0.smithi169.stderr: 2022-03-01T06:31:34.619 INFO:tasks.workunit.client.0.smithi169.stderr:HEAD is now at 45ece157fc2 Merge branch 'wip-54377-octopus' of https://github.com/pponnuvel/ceph into wip-yuri11-testing-2022-02-28-0803-octopus 2022-03-01T06:31:34.620 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-03-01T06:31:34.737 DEBUG:teuthology.orchestra.remote:smithi169:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2022-03-01T06:31:34.786 INFO:tasks.workunit:Running workunits matching osd on client.0... 2022-03-01T06:31:34.786 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2022-03-01T06:31:34.787 DEBUG:teuthology.orchestra.run.smithi169:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=45ece157fc29b55a5ee952c81c0c23abdb89bf98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2022-03-01T06:31:34.818 INFO:tasks.workunit.client.0.smithi169.stderr:stty: 'standard input': Inappropriate ioctl for device 2022-03-01T06:31:34.848 INFO:tasks.workunit.client.0.smithi169.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2022-03-01T06:31:34.849 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2022-03-01T06:31:34.849 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2022-03-01T06:31:34.849 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2022-03-01T06:31:34.850 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2022-03-01T06:31:34.850 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2022-03-01T06:31:34.850 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2022-03-01T06:31:34.850 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2022-03-01T06:31:34.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2022-03-01T06:31:34.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/bad-inc-map 2022-03-01T06:31:34.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2022-03-01T06:31:34.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2022-03-01T06:31:34.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:42115 2022-03-01T06:31:34.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:42115 2022-03-01T06:31:34.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2022-03-01T06:31:34.852 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2022-03-01T06:31:34.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none ' 2022-03-01T06:31:34.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:42115 ' 2022-03-01T06:31:34.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2022-03-01T06:31:34.855 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2022-03-01T06:31:34.856 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2022-03-01T06:31:34.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2022-03-01T06:31:34.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2022-03-01T06:31:34.857 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2022-03-01T06:31:34.857 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2022-03-01T06:31:34.857 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2022-03-01T06:31:34.858 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2022-03-01T06:31:34.858 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:31:34.858 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2022-03-01T06:31:34.858 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:31:34.859 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:31:34.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:31:34.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:31:34.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:31:34.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:31:34.862 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:31:34.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:31:34.863 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:31:34.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:31:34.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:31:34.863 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:31:34.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:31:34.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:31:34.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:31:34.864 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:31:34.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:31:34.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:31:34.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2022-03-01T06:31:34.865 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:31:34.866 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:34.866 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:34.866 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29813 2022-03-01T06:31:34.869 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:31:34.869 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:31:34.869 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2022-03-01T06:31:34.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-03-01T06:31:34.870 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:34.870 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:34.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29813 2022-03-01T06:31:34.870 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-03-01T06:31:34.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2022-03-01T06:31:34.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2022-03-01T06:31:34.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-03-01T06:31:34.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2022-03-01T06:31:34.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2022-03-01T06:31:34.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2022-03-01T06:31:34.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2022-03-01T06:31:34.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2022-03-01T06:31:34.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-03-01T06:31:34.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-03-01T06:31:34.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-03-01T06:31:34.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2022-03-01T06:31:34.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2022-03-01T06:31:35.182 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-03-01T06:31:35.182 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:31:35.182 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:31:35.183 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:31:35.183 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:35.183 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:35.183 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:35.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-03-01T06:31:35.341 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-03-01T06:31:35.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-03-01T06:31:35.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:31:35.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:31:35.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-03-01T06:31:35.344 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-03-01T06:31:35.344 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:31:35.344 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:31:35.344 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:31:35.345 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:31:35.345 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:35.345 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:35.345 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29813/ceph-mon.a.asok 2022-03-01T06:31:35.345 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:31:35.346 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.29813/ceph-mon.a.asok config get fsid 2022-03-01T06:31:35.427 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-03-01T06:31:35.428 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:31:35.428 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:31:35.428 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-03-01T06:31:35.428 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-03-01T06:31:35.429 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:31:35.429 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:31:35.429 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:31:35.429 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:31:35.429 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:35.430 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:35.430 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29813/ceph-mon.a.asok 2022-03-01T06:31:35.430 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:31:35.430 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.29813/ceph-mon.a.asok config get mon_host 2022-03-01T06:31:35.511 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2022-03-01T06:31:35.511 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/bad-inc-map 2022-03-01T06:31:35.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-03-01T06:31:35.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-03-01T06:31:35.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-03-01T06:31:35.513 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/bad-inc-map/x 2022-03-01T06:31:35.513 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-03-01T06:31:36.356 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-03-01T06:31:36.356 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:31:36.357 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:31:36.357 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:31:36.357 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:36.358 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:36.358 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:36.358 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:31:36.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:31:36.381 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2022-03-01T06:31:36.382 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2022-03-01T06:31:36.382 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:31:36.382 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-03-01T06:31:36.383 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:31:36.383 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/0 2022-03-01T06:31:36.383 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 ' 2022-03-01T06:31:36.383 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:31:36.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:31:36.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:31:36.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2022-03-01T06:31:36.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2022-03-01T06:31:36.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:31:36.385 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:31:36.385 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2022-03-01T06:31:36.385 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:31:36.385 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:31:36.386 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:31:36.386 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:31:36.386 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:36.386 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:36.387 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:36.387 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:36.387 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:31:36.387 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:31:36.387 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:31:36.388 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2022-03-01T06:31:36.388 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2022-03-01T06:31:36.388 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:31:36.388 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:31:36.389 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:31:36.389 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:31:36.389 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:31:36.389 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/0 2022-03-01T06:31:36.390 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:31:36.390 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fc8b8934-98a7-4ba1-8994-c2a508befca7 2022-03-01T06:31:36.390 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 fc8b8934-98a7-4ba1-8994-c2a508befca7' 2022-03-01T06:31:36.391 INFO:tasks.workunit.client.0.smithi169.stdout:add osd0 fc8b8934-98a7-4ba1-8994-c2a508befca7 2022-03-01T06:31:36.391 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:31:36.392 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDIvR1i8XCBGBAAPOvrwz/8CPqVqVEcfvW3MQ== 2022-03-01T06:31:36.392 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDIvR1i8XCBGBAAPOvrwz/8CPqVqVEcfvW3MQ=="}' 2022-03-01T06:31:36.393 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fc8b8934-98a7-4ba1-8994-c2a508befca7 -i td/bad-inc-map/0/new.json 2022-03-01T06:31:36.685 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:31:36.695 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/0/new.json 2022-03-01T06:31:36.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDIvR1i8XCBGBAAPOvrwz/8CPqVqVEcfvW3MQ== --osd-uuid fc8b8934-98a7-4ba1-8994-c2a508befca7 2022-03-01T06:31:36.716 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.735+0000 7f05ba575f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:36.718 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.737+0000 7f05ba575f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:36.720 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.737+0000 7f05ba575f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:36.720 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.738+0000 7f05ba575f00 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2022-03-01T06:31:36.721 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.738+0000 7f05ba575f00 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2022-03-01T06:31:36.721 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.738+0000 7f05ba575f00 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2022-03-01T06:31:36.782 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:36.800+0000 7f05ba575f00 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2022-03-01T06:31:37.277 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:37.295+0000 7f05ba575f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:31:39.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/0/keyring 2022-03-01T06:31:39.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:31:39.838 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-03-01T06:31:39.838 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd0 key to auth repository 2022-03-01T06:31:39.840 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:31:40.043 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:31:40.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-03-01T06:31:40.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:31:40.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:31:40.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:31:40.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:31:40.063 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:40.081+0000 7f74c22c2f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:40.074 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:40.092+0000 7f74c22c2f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:40.075 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:40.093+0000 7f74c22c2f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:40.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-03-01T06:31:40.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:31:40.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:31:40.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:31:40.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:31:40.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:40.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:31:40.245 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:31:40.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:40.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:31:40.439 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:31:40.608 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:40.619+0000 7f74c22c2f00 -1 Falling back to public interface 2022-03-01T06:31:41.367 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:41.385+0000 7f74c22c2f00 -1 osd.0 0 log_to_monitors {default=true} 2022-03-01T06:31:41.440 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:31:41.441 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:41.441 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:31:41.442 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:31:41.442 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:41.442 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:31:41.792 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:31:42.789 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:31:42.790 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:42.790 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:31:42.791 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-03-01T06:31:42.791 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:42.791 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:31:43.163 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/30185,v1:127.0.0.1:6801/30185] [v2:127.0.0.1:6802/30185,v1:127.0.0.1:6803/30185] exists,up fc8b8934-98a7-4ba1-8994-c2a508befca7 2022-03-01T06:31:43.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:31:43.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:31:43.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:31:43.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2022-03-01T06:31:43.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2022-03-01T06:31:43.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:31:43.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-03-01T06:31:43.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:31:43.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/1 2022-03-01T06:31:43.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 ' 2022-03-01T06:31:43.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:31:43.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:31:43.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:31:43.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2022-03-01T06:31:43.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2022-03-01T06:31:43.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:31:43.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:31:43.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2022-03-01T06:31:43.171 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:31:43.172 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:31:43.172 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:31:43.172 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:31:43.172 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:43.172 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:43.173 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:43.173 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:43.173 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:31:43.173 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:31:43.173 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:31:43.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2022-03-01T06:31:43.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2022-03-01T06:31:43.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:31:43.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:31:43.175 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:31:43.175 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:31:43.175 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:31:43.175 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/1 2022-03-01T06:31:43.175 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:31:43.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d848ff61-cbf8-4bfe-a9c7-85447748c6d1 2022-03-01T06:31:43.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 d848ff61-cbf8-4bfe-a9c7-85447748c6d1' 2022-03-01T06:31:43.176 INFO:tasks.workunit.client.0.smithi169.stdout:add osd1 d848ff61-cbf8-4bfe-a9c7-85447748c6d1 2022-03-01T06:31:43.177 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:31:43.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDPvR1iehfLCxAAPtcAMteToA7r9VkiS1bY/g== 2022-03-01T06:31:43.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDPvR1iehfLCxAAPtcAMteToA7r9VkiS1bY/g=="}' 2022-03-01T06:31:43.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d848ff61-cbf8-4bfe-a9c7-85447748c6d1 -i td/bad-inc-map/1/new.json 2022-03-01T06:31:43.668 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:31:43.678 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/1/new.json 2022-03-01T06:31:43.679 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDPvR1iehfLCxAAPtcAMteToA7r9VkiS1bY/g== --osd-uuid d848ff61-cbf8-4bfe-a9c7-85447748c6d1 2022-03-01T06:31:43.699 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.718+0000 7f626dd6ff00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:43.701 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.720+0000 7f626dd6ff00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:43.703 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.721+0000 7f626dd6ff00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:43.704 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.721+0000 7f626dd6ff00 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2022-03-01T06:31:43.704 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.721+0000 7f626dd6ff00 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2022-03-01T06:31:43.705 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.721+0000 7f626dd6ff00 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2022-03-01T06:31:43.774 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:43.792+0000 7f626dd6ff00 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2022-03-01T06:31:44.167 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:44.186+0000 7f626dd6ff00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:31:46.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/1/keyring 2022-03-01T06:31:46.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:31:46.723 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-03-01T06:31:46.723 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd1 key to auth repository 2022-03-01T06:31:46.724 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:31:47.154 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:31:47.155 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-03-01T06:31:47.156 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:31:47.156 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:31:47.157 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:31:47.157 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:31:47.173 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:47.192+0000 7f2a34bb2f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:47.175 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:47.194+0000 7f2a34bb2f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:47.176 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:47.195+0000 7f2a34bb2f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:47.487 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-03-01T06:31:47.487 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:31:47.487 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:31:47.487 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:31:47.488 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:31:47.488 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:47.488 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:31:47.489 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:31:47.489 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:47.489 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:31:47.696 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:47.715+0000 7f2a34bb2f00 -1 Falling back to public interface 2022-03-01T06:31:47.803 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:31:48.431 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:48.450+0000 7f2a34bb2f00 -1 osd.1 0 log_to_monitors {default=true} 2022-03-01T06:31:48.804 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:31:48.804 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:48.805 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:31:48.805 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:31:48.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:48.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:31:49.197 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/30636,v1:127.0.0.1:6811/30636] [v2:127.0.0.1:6812/30636,v1:127.0.0.1:6813/30636] exists,up d848ff61-cbf8-4bfe-a9c7-85447748c6d1 2022-03-01T06:31:49.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:31:49.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:31:49.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:31:49.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2022-03-01T06:31:49.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2022-03-01T06:31:49.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:31:49.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-03-01T06:31:49.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:31:49.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/2 2022-03-01T06:31:49.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 ' 2022-03-01T06:31:49.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:31:49.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:31:49.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:31:49.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2022-03-01T06:31:49.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2022-03-01T06:31:49.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:31:49.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:31:49.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2022-03-01T06:31:49.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:31:49.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:31:49.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:31:49.207 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:31:49.207 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:31:49.207 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:31:49.207 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:49.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' 2022-03-01T06:31:49.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:31:49.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:31:49.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:31:49.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2022-03-01T06:31:49.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2022-03-01T06:31:49.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:31:49.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:31:49.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:31:49.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:31:49.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:31:49.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/2 2022-03-01T06:31:49.210 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:31:49.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=eacf5fa2-1441-4283-bf48-c4e684f597aa 2022-03-01T06:31:49.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 eacf5fa2-1441-4283-bf48-c4e684f597aa' 2022-03-01T06:31:49.211 INFO:tasks.workunit.client.0.smithi169.stdout:add osd2 eacf5fa2-1441-4283-bf48-c4e684f597aa 2022-03-01T06:31:49.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:31:49.213 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDVvR1iaTrfDRAA3/41zO3IWd7eUE/BQHmJyw== 2022-03-01T06:31:49.213 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDVvR1iaTrfDRAA3/41zO3IWd7eUE/BQHmJyw=="}' 2022-03-01T06:31:49.214 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new eacf5fa2-1441-4283-bf48-c4e684f597aa -i td/bad-inc-map/2/new.json 2022-03-01T06:31:49.706 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:31:49.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/2/new.json 2022-03-01T06:31:49.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVvR1iaTrfDRAA3/41zO3IWd7eUE/BQHmJyw== --osd-uuid eacf5fa2-1441-4283-bf48-c4e684f597aa 2022-03-01T06:31:49.738 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.756+0000 7fc54105ef00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:49.764 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.783+0000 7fc54105ef00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:49.765 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.784+0000 7fc54105ef00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:49.766 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.784+0000 7fc54105ef00 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2022-03-01T06:31:49.766 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.784+0000 7fc54105ef00 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2022-03-01T06:31:49.766 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.784+0000 7fc54105ef00 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2022-03-01T06:31:49.840 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:49.859+0000 7fc54105ef00 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2022-03-01T06:31:50.200 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:50.219+0000 7fc54105ef00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:31:52.770 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/2/keyring 2022-03-01T06:31:52.770 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:31:52.771 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd2 key to auth repository 2022-03-01T06:31:52.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-03-01T06:31:52.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:31:53.199 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-03-01T06:31:53.199 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:31:53.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:31:53.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=ad099fa4-b875-4a0a-ac34-43fb95a0f08a --auth-supported=none --mon-host=127.0.0.1:42115 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:31:53.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:31:53.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:31:53.218 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:53.236+0000 7f43bba2af00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:53.219 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:53.238+0000 7f43bba2af00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:53.220 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:53.239+0000 7f43bba2af00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:31:53.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-03-01T06:31:53.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:31:53.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:31:53.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:31:53.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:31:53.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:53.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:31:53.526 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:31:53.526 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:53.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:31:53.740 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:53.759+0000 7f43bba2af00 -1 Falling back to public interface 2022-03-01T06:31:53.841 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:31:54.522 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:31:54.541+0000 7f43bba2af00 -1 osd.2 0 log_to_monitors {default=true} 2022-03-01T06:31:54.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:31:54.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:31:54.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:31:54.843 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:31:54.844 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:31:54.844 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:31:55.247 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/30991,v1:127.0.0.1:6819/30991] [v2:127.0.0.1:6820/30991,v1:127.0.0.1:6821/30991] exists,up eacf5fa2-1441-4283-bf48-c4e684f597aa 2022-03-01T06:31:55.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:31:55.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:31:55.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:31:55.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2022-03-01T06:31:55.637 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2022-03-01T06:31:55.638 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 8 2022-03-01T06:31:56.115 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'foo' already exists 2022-03-01T06:31:56.123 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-03-01T06:31:57.124 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2022-03-01T06:31:57.912 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 min_size to 1 2022-03-01T06:31:57.920 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2022-03-01T06:31:59.068 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 min_size to 2 2022-03-01T06:31:59.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2022-03-01T06:32:04.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2022-03-01T06:32:04.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2022-03-01T06:32:04.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:32:04.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:32:04.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:32:04.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:32:04.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2022-03-01T06:32:04.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:32:04.081 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:04.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:04.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:32:04.744 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/30185,v1:127.0.0.1:6801/30185] [v2:127.0.0.1:6802/30185,v1:127.0.0.1:6803/30185] exists,up fc8b8934-98a7-4ba1-8994-c2a508befca7 2022-03-01T06:32:04.745 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:32:04.746 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:32:04.746 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:32:04.746 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2022-03-01T06:32:04.746 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2022-03-01T06:32:04.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:32:04.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:32:04.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:32:04.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:32:04.748 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2022-03-01T06:32:04.748 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:32:04.749 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:04.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:04.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:32:05.118 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/30636,v1:127.0.0.1:6811/30636] [v2:127.0.0.1:6812/30636,v1:127.0.0.1:6813/30636] exists,up d848ff61-cbf8-4bfe-a9c7-85447748c6d1 2022-03-01T06:32:05.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:32:05.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:32:05.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:32:05.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2022-03-01T06:32:05.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2022-03-01T06:32:05.122 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:32:05.122 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:32:05.123 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:32:05.123 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:32:05.124 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2022-03-01T06:32:05.124 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:32:05.125 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:05.125 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:05.126 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:32:05.499 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/30991,v1:127.0.0.1:6819/30991] [v2:127.0.0.1:6820/30991,v1:127.0.0.1:6821/30991] exists,up eacf5fa2-1441-4283-bf48-c4e684f597aa 2022-03-01T06:32:05.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:32:05.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:32:05.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:32:05.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2022-03-01T06:32:05.501 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:56.041+0000 7f43a7ad3700 -1 osd.2 15 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.501 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:56.344+0000 7f43a7ad3700 -1 osd.2 17 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.501 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:56.670+0000 7f43a7ad3700 -1 osd.2 18 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.502 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:57.443+0000 7f43a290d700 -1 osd.2 19 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.502 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:57.444+0000 7f43a290d700 -1 osd.2 19 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.502 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:57.677+0000 7f43a7ad3700 -1 osd.2 20 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.502 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.146+0000 7f43a290d700 -1 osd.2 22 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.503 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.147+0000 7f43a290d700 -1 osd.2 22 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.503 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.247+0000 7f43a7ad3700 -1 osd.2 22 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.503 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.526+0000 7f43a7ad3700 -1 osd.2 24 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.503 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.741+0000 7f43a7ad3700 -1 osd.2 24 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.503 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:59.286+0000 7f43a290d700 -1 osd.2 26 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.504 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:59.370+0000 7f43a7ad3700 -1 osd.2 26 handle_osd_map injecting map crc failure 2022-03-01T06:32:05.504 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2022-03-01T06:32:05.505 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:56.041+0000 7f43a7ad3700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2022-03-01T06:32:05.505 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:56.344+0000 7f43a7ad3700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2022-03-01T06:32:05.505 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:56.670+0000 7f43a7ad3700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2022-03-01T06:32:05.505 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:57.444+0000 7f43a290d700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2022-03-01T06:32:05.506 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:57.444+0000 7f43a290d700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2022-03-01T06:32:05.506 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:57.677+0000 7f43a7ad3700 10 osd.2 20 handle_osd_map bailing because last < start (21<22) 2022-03-01T06:32:05.506 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.147+0000 7f43a290d700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2022-03-01T06:32:05.506 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.147+0000 7f43a290d700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2022-03-01T06:32:05.507 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.248+0000 7f43a7ad3700 10 osd.2 22 handle_osd_map bailing because last < start (23<24) 2022-03-01T06:32:05.507 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.527+0000 7f43a7ad3700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2022-03-01T06:32:05.507 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:58.742+0000 7f43a7ad3700 10 osd.2 24 handle_osd_map bailing because last < start (25<26) 2022-03-01T06:32:05.507 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:59.286+0000 7f43a290d700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2022-03-01T06:32:05.507 INFO:tasks.workunit.client.0.smithi169.stdout:2022-03-01T06:31:59.371+0000 7f43a7ad3700 10 osd.2 26 handle_osd_map bailing because last < start (27<28) 2022-03-01T06:32:05.508 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2022-03-01T06:32:05.508 INFO:tasks.workunit.client.0.smithi169.stdout:success 2022-03-01T06:32:05.509 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2022-03-01T06:32:05.509 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=foo 2022-03-01T06:32:05.509 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2022-03-01T06:32:06.228 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'foo' does not exist 2022-03-01T06:32:06.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2022-03-01T06:32:06.237 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:32:06.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:32:06.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:32:06.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:32:06.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:32:06.345 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:32:06.345 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2022-03-01T06:32:06.346 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2022-03-01T06:32:06.346 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:32:06.346 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2022-03-01T06:32:06.346 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:32:06.347 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:32:06.347 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:32:06.347 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:32:06.347 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:32:06.349 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:32:06.350 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:32:06.350 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:32:06.351 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:32:06.351 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:32:06.352 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:32:06.352 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:32:06.353 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:32:06.354 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:32:06.354 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:32:06.354 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:32:06.355 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:32:06.355 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:32:06.356 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2022-03-01T06:32:06.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:32:06.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:06.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:32:06.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29813 2022-03-01T06:32:06.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:32:06.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:32:06.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2022-03-01T06:32:06.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/bad-inc-map 0 2022-03-01T06:32:06.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2022-03-01T06:32:06.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2022-03-01T06:32:06.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2022-03-01T06:32:06.365 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:32:06.365 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:32:06.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:32:06.366 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:32:06.366 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:32:06.367 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:32:06.367 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:32:06.368 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:32:06.368 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:32:06.368 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:32:06.368 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:32:06.369 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:32:06.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:32:06.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:32:06.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:32:06.370 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:32:06.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:32:06.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2022-03-01T06:32:06.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2022-03-01T06:32:06.373 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:32:06.373 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:06.374 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29813 2022-03-01T06:32:06.374 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29813 2022-03-01T06:32:06.374 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:32:06.374 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:32:06.375 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2022-03-01T06:32:06.375 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-03-01T06:32:06.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-03-01T06:32:06.440 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2022-03-01T06:32:06.441 DEBUG:teuthology.orchestra.run.smithi169:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=45ece157fc29b55a5ee952c81c0c23abdb89bf98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2022-03-01T06:32:06.503 INFO:tasks.workunit.client.0.smithi169.stderr:stty: 'standard input': Inappropriate ioctl for device 2022-03-01T06:32:06.513 INFO:tasks.workunit.client.0.smithi169.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2022-03-01T06:32:06.513 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2022-03-01T06:32:06.514 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2022-03-01T06:32:06.514 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2022-03-01T06:32:06.514 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2022-03-01T06:32:06.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2022-03-01T06:32:06.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2022-03-01T06:32:06.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2022-03-01T06:32:06.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2022-03-01T06:32:06.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/divergent-priors 2022-03-01T06:32:06.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2022-03-01T06:32:06.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2022-03-01T06:32:06.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2022-03-01T06:32:06.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2022-03-01T06:32:06.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2022-03-01T06:32:06.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2022-03-01T06:32:06.517 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2022-03-01T06:32:06.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2022-03-01T06:32:06.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2022-03-01T06:32:06.518 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2022-03-01T06:32:06.518 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2022-03-01T06:32:06.518 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2022-03-01T06:32:06.518 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2022-03-01T06:32:06.523 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2022-03-01T06:32:06.523 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2022-03-01T06:32:06.524 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2022-03-01T06:32:06.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2022-03-01T06:32:06.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2022-03-01T06:32:06.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2022-03-01T06:32:06.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2022-03-01T06:32:06.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2022-03-01T06:32:06.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2022-03-01T06:32:06.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2022-03-01T06:32:06.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2022-03-01T06:32:06.526 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2022-03-01T06:32:06.526 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2022-03-01T06:32:06.526 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none ' 2022-03-01T06:32:06.526 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2022-03-01T06:32:06.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2022-03-01T06:32:06.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2022-03-01T06:32:06.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:32:06.527 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2022-03-01T06:32:06.527 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2022-03-01T06:32:06.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2022-03-01T06:32:06.528 INFO:tasks.workunit.client.0.smithi169.stderr:TEST_divergent_2 2022-03-01T06:32:06.528 INFO:tasks.workunit.client.0.smithi169.stderr:TEST_divergent_3 2022-03-01T06:32:06.528 INFO:tasks.workunit.client.0.smithi169.stderr:TEST_divergent_ec' 2022-03-01T06:32:06.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2022-03-01T06:32:06.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2022-03-01T06:32:06.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2022-03-01T06:32:06.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2022-03-01T06:32:06.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-03-01T06:32:06.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:32:06.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-03-01T06:32:06.530 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:32:06.530 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:32:06.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:32:06.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:32:06.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:32:06.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:32:06.532 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:32:06.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:32:06.532 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:32:06.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:32:06.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:32:06.533 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:32:06.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:32:06.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:32:06.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:32:06.537 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:32:06.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:32:06.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:32:06.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-03-01T06:32:06.538 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:32:06.538 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:06.539 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:06.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31520 2022-03-01T06:32:06.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:32:06.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:32:06.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2022-03-01T06:32:06.540 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-03-01T06:32:06.540 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:06.540 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:06.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.31520 2022-03-01T06:32:06.541 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-03-01T06:32:06.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2022-03-01T06:32:06.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2022-03-01T06:32:06.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-03-01T06:32:06.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2022-03-01T06:32:06.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2022-03-01T06:32:06.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2022-03-01T06:32:06.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2022-03-01T06:32:06.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2022-03-01T06:32:06.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2022-03-01T06:32:06.543 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2022-03-01T06:32:06.544 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2022-03-01T06:32:06.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2022-03-01T06:32:06.545 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:32:06.545 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:32:06.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2022-03-01T06:32:06.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2022-03-01T06:32:06.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-03-01T06:32:06.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-03-01T06:32:06.547 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-03-01T06:32:06.547 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2022-03-01T06:32:06.547 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2022-03-01T06:32:06.840 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-03-01T06:32:06.841 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:32:06.841 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:32:06.841 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:32:06.842 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:06.842 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:06.842 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:06.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-03-01T06:32:07.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-03-01T06:32:07.002 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-03-01T06:32:07.002 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:32:07.002 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:32:07.003 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-03-01T06:32:07.003 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-03-01T06:32:07.003 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:32:07.003 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:32:07.004 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:32:07.004 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:32:07.004 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:07.004 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:07.005 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-mon.a.asok 2022-03-01T06:32:07.005 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:32:07.005 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.31520/ceph-mon.a.asok config get fsid 2022-03-01T06:32:07.088 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-03-01T06:32:07.088 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:32:07.088 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:32:07.089 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-03-01T06:32:07.089 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-03-01T06:32:07.089 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:32:07.089 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:32:07.090 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:32:07.090 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:32:07.090 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:07.090 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:07.091 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-mon.a.asok 2022-03-01T06:32:07.091 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:32:07.091 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.31520/ceph-mon.a.asok config get mon_host 2022-03-01T06:32:07.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2022-03-01T06:32:07.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2022-03-01T06:32:07.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-03-01T06:32:07.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-03-01T06:32:07.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-03-01T06:32:07.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2022-03-01T06:32:07.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-03-01T06:32:08.042 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-03-01T06:32:08.042 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:32:08.042 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:32:08.043 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:32:08.043 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:08.043 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:08.043 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:08.044 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:32:08.044 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:32:08.073 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2022-03-01T06:32:08.073 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2022-03-01T06:32:08.074 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:32:08.074 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:32:08.074 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-03-01T06:32:08.074 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:32:08.075 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2022-03-01T06:32:08.075 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:32:08.075 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:32:08.075 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:32:08.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:32:08.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-03-01T06:32:08.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-03-01T06:32:08.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:32:08.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:32:08.077 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:32:08.077 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:32:08.077 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:32:08.077 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:32:08.078 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:32:08.078 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:08.078 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:08.078 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:08.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:08.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:32:08.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:32:08.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:32:08.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:32:08.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:32:08.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:32:08.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:32:08.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:32:08.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:32:08.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:32:08.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2022-03-01T06:32:08.081 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:32:08.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b36cc591-5c05-4344-a0c6-fd7c20be6391 2022-03-01T06:32:08.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 b36cc591-5c05-4344-a0c6-fd7c20be6391' 2022-03-01T06:32:08.082 INFO:tasks.workunit.client.0.smithi169.stdout:add osd0 b36cc591-5c05-4344-a0c6-fd7c20be6391 2022-03-01T06:32:08.083 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:32:08.092 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDovR1iN36hBhAAncUyg+0+TPr4OVYl9Sx3rg== 2022-03-01T06:32:08.092 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDovR1iN36hBhAAncUyg+0+TPr4OVYl9Sx3rg=="}' 2022-03-01T06:32:08.092 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b36cc591-5c05-4344-a0c6-fd7c20be6391 -i td/divergent-priors/0/new.json 2022-03-01T06:32:08.404 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:08.414 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2022-03-01T06:32:08.415 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDovR1iN36hBhAAncUyg+0+TPr4OVYl9Sx3rg== --osd-uuid b36cc591-5c05-4344-a0c6-fd7c20be6391 2022-03-01T06:32:08.444 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.463+0000 7fe716bd5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:08.446 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.465+0000 7fe716bd5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:08.447 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.466+0000 7fe716bd5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:08.448 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.466+0000 7fe716bd5f00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:32:08.448 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.466+0000 7fe716bd5f00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:32:08.448 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.466+0000 7fe716bd5f00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:32:08.490 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.509+0000 7fe716bd5f00 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2022-03-01T06:32:08.851 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:08.869+0000 7fe716bd5f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:32:11.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2022-03-01T06:32:11.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:32:11.480 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-03-01T06:32:11.481 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd0 key to auth repository 2022-03-01T06:32:11.481 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:32:11.913 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:32:11.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-03-01T06:32:11.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:32:11.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:32:11.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:32:11.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:32:11.941 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:11.959+0000 7fad46c3bf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:11.942 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:11.961+0000 7fad46c3bf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:11.943 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:11.962+0000 7fad46c3bf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:12.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-03-01T06:32:12.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:32:12.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:32:12.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:32:12.259 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:32:12.259 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:32:12.259 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:32:12.260 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:12.260 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:12.260 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:32:12.463 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:12.482+0000 7fad46c3bf00 -1 Falling back to public interface 2022-03-01T06:32:12.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:32:13.192 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:13.211+0000 7fad46c3bf00 -1 osd.0 0 log_to_monitors {default=true} 2022-03-01T06:32:13.584 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:32:13.584 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:32:13.584 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:32:13.585 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:32:13.585 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:13.586 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:32:13.981 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31953,v1:127.0.0.1:6803/31953] [v2:127.0.0.1:6804/31953,v1:127.0.0.1:6805/31953] exists,up b36cc591-5c05-4344-a0c6-fd7c20be6391 2022-03-01T06:32:13.986 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:32:13.987 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:32:13.987 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:32:13.987 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2022-03-01T06:32:13.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2022-03-01T06:32:13.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:32:13.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:32:13.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-03-01T06:32:13.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:32:13.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:32:13.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:32:13.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:32:13.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:32:13.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:32:13.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:32:13.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:32:13.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:32:13.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:32:13.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:32:13.991 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:32:13.991 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:32:13.992 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:32:13.992 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:32:13.992 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:13.992 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:13.993 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:13.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:13.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:32:13.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:32:13.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:32:13.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:32:13.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:32:13.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:32:13.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:32:13.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:32:13.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:32:13.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:32:13.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:32:13.996 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:32:13.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fd55fbe2-fa36-4813-92e3-6da6f78c70a0 2022-03-01T06:32:13.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 fd55fbe2-fa36-4813-92e3-6da6f78c70a0' 2022-03-01T06:32:13.996 INFO:tasks.workunit.client.0.smithi169.stdout:add osd1 fd55fbe2-fa36-4813-92e3-6da6f78c70a0 2022-03-01T06:32:13.997 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:32:14.003 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDuvR1iWd1fARAA+kB/wgCrW/G1Btn9svh4/g== 2022-03-01T06:32:14.004 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDuvR1iWd1fARAA+kB/wgCrW/G1Btn9svh4/g=="}' 2022-03-01T06:32:14.004 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fd55fbe2-fa36-4813-92e3-6da6f78c70a0 -i td/divergent-priors/1/new.json 2022-03-01T06:32:14.436 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:32:14.446 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2022-03-01T06:32:14.446 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDuvR1iWd1fARAA+kB/wgCrW/G1Btn9svh4/g== --osd-uuid fd55fbe2-fa36-4813-92e3-6da6f78c70a0 2022-03-01T06:32:14.476 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.494+0000 7f08ce350f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:14.477 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.496+0000 7f08ce350f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:14.478 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.497+0000 7f08ce350f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:14.479 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.497+0000 7f08ce350f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:32:14.479 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.497+0000 7f08ce350f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:32:14.479 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.497+0000 7f08ce350f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:32:14.558 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:14.577+0000 7f08ce350f00 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2022-03-01T06:32:14.985 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:15.004+0000 7f08ce350f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:32:17.605 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2022-03-01T06:32:17.605 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:32:17.607 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-03-01T06:32:17.608 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd1 key to auth repository 2022-03-01T06:32:17.608 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:32:18.039 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-03-01T06:32:18.040 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:32:18.041 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:32:18.042 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:32:18.042 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:32:18.043 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:32:18.067 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:18.085+0000 7f22c4a49f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:18.068 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:18.087+0000 7f22c4a49f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:18.069 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:18.088+0000 7f22c4a49f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:18.389 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-03-01T06:32:18.390 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:32:18.390 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:32:18.390 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:32:18.390 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:32:18.391 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:32:18.391 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:32:18.391 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:18.392 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:18.392 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:32:18.584 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:18.603+0000 7f22c4a49f00 -1 Falling back to public interface 2022-03-01T06:32:18.710 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:32:19.316 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:19.335+0000 7f22c4a49f00 -1 osd.1 0 log_to_monitors {default=true} 2022-03-01T06:32:19.711 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:32:19.712 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:32:19.712 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:32:19.712 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:32:19.713 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:19.713 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:32:20.047 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/32308,v1:127.0.0.1:6811/32308] [v2:127.0.0.1:6812/32308,v1:127.0.0.1:6813/32308] exists,up fd55fbe2-fa36-4813-92e3-6da6f78c70a0 2022-03-01T06:32:20.051 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:32:20.052 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:32:20.052 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:32:20.052 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2022-03-01T06:32:20.053 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2022-03-01T06:32:20.053 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:32:20.053 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:32:20.053 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-03-01T06:32:20.054 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:32:20.054 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2022-03-01T06:32:20.054 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:32:20.054 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:32:20.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:32:20.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:32:20.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-03-01T06:32:20.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-03-01T06:32:20.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:32:20.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:32:20.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:32:20.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:32:20.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:32:20.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:32:20.057 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:32:20.058 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:20.058 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:20.058 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:20.058 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:32:20.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:32:20.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:32:20.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:32:20.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:32:20.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:32:20.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:32:20.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:32:20.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:32:20.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:32:20.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:32:20.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2022-03-01T06:32:20.061 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:32:20.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9d15391e-aa51-4a5b-868e-961f98e20902 2022-03-01T06:32:20.062 INFO:tasks.workunit.client.0.smithi169.stdout:add osd2 9d15391e-aa51-4a5b-868e-961f98e20902 2022-03-01T06:32:20.062 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 9d15391e-aa51-4a5b-868e-961f98e20902' 2022-03-01T06:32:20.063 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:32:20.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD0vR1inFpIBRAAnhVvZZST2ixcHY3akl6w9w== 2022-03-01T06:32:20.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD0vR1inFpIBRAAnhVvZZST2ixcHY3akl6w9w=="}' 2022-03-01T06:32:20.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9d15391e-aa51-4a5b-868e-961f98e20902 -i td/divergent-priors/2/new.json 2022-03-01T06:32:20.510 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:32:20.520 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2022-03-01T06:32:20.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD0vR1inFpIBRAAnhVvZZST2ixcHY3akl6w9w== --osd-uuid 9d15391e-aa51-4a5b-868e-961f98e20902 2022-03-01T06:32:20.551 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.569+0000 7fea0e319f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:20.552 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.571+0000 7fea0e319f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:20.553 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.572+0000 7fea0e319f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:20.554 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.572+0000 7fea0e319f00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:32:20.554 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.573+0000 7fea0e319f00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:32:20.554 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.573+0000 7fea0e319f00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:32:20.591 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.610+0000 7fea0e319f00 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2022-03-01T06:32:20.968 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:20.987+0000 7fea0e319f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:32:23.595 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2022-03-01T06:32:23.595 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:32:23.596 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-03-01T06:32:23.597 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd2 key to auth repository 2022-03-01T06:32:23.597 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:32:24.031 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-03-01T06:32:24.032 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:32:24.033 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:32:24.033 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:32:24.034 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:32:24.034 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:32:24.059 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:24.078+0000 7f3b982e4f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:24.061 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:24.080+0000 7f3b982e4f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:24.062 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:24.081+0000 7f3b982e4f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:32:24.361 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-03-01T06:32:24.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:32:24.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:32:24.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:32:24.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:32:24.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:32:24.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:32:24.364 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:32:24.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:24.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:32:24.578 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:24.597+0000 7f3b982e4f00 -1 Falling back to public interface 2022-03-01T06:32:24.687 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:32:25.300 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:32:25.318+0000 7f3b982e4f00 -1 osd.2 0 log_to_monitors {default=true} 2022-03-01T06:32:25.688 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:32:25.689 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:32:25.689 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:32:25.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:32:25.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:32:25.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:32:26.058 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/32663,v1:127.0.0.1:6819/32663] [v2:127.0.0.1:6820/32663,v1:127.0.0.1:6821/32663] exists,up 9d15391e-aa51-4a5b-868e-961f98e20902 2022-03-01T06:32:26.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:32:26.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:32:26.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:32:26.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2022-03-01T06:32:26.870 INFO:tasks.workunit.client.0.smithi169.stderr:noout is set 2022-03-01T06:32:26.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2022-03-01T06:32:27.572 INFO:tasks.workunit.client.0.smithi169.stderr:noin is set 2022-03-01T06:32:27.580 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2022-03-01T06:32:28.223 INFO:tasks.workunit.client.0.smithi169.stderr:nodown is set 2022-03-01T06:32:28.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2022-03-01T06:32:28.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2022-03-01T06:32:28.713 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'test' already exists 2022-03-01T06:32:28.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-03-01T06:32:29.723 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2022-03-01T06:32:30.373 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 size to 3 2022-03-01T06:32:30.381 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2022-03-01T06:32:30.967 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 min_size to 2 2022-03-01T06:32:30.975 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2022-03-01T06:32:30.975 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:32:30.976 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:32:31.305 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:32:31.305 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:32:31.306 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:32:31.306 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:32:31.306 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:31.306 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:32:31.430 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2022-03-01T06:32:31.430 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2022-03-01T06:32:31.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2022-03-01T06:32:31.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:31.431 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:32:31.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705669 2022-03-01T06:32:31.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705669 2022-03-01T06:32:31.555 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2022-03-01T06:32:31.555 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:31.555 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:32:31.678 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574851 2022-03-01T06:32:31.678 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574851 2022-03-01T06:32:31.678 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-55834574851' 2022-03-01T06:32:31.679 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:31.679 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2022-03-01T06:32:31.679 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:31.680 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:32:31.681 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2022-03-01T06:32:31.681 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:31.681 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2022-03-01T06:32:31.682 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2022-03-01T06:32:31.683 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836487 2022-03-01T06:32:31.683 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:32:32.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2022-03-01T06:32:32.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:32:33.001 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:32:33.002 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:32:33.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2022-03-01T06:32:33.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:32:34.337 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 299 -eq 0 ']' 2022-03-01T06:32:34.338 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:32:34.674 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2022-03-01T06:32:34.675 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:34.675 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705669 2022-03-01T06:32:34.675 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:34.676 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:32:34.676 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705669 2022-03-01T06:32:34.676 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:34.677 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705669 2022-03-01T06:32:34.677 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2022-03-01T06:32:34.678 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 38654705669 2022-03-01T06:32:34.678 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:32:35.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705669 -lt 38654705669 2022-03-01T06:32:35.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:35.000 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-55834574851 2022-03-01T06:32:35.001 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:35.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:32:35.002 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-55834574851 2022-03-01T06:32:35.002 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:35.003 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574851 2022-03-01T06:32:35.003 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2022-03-01T06:32:35.004 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 55834574851 2022-03-01T06:32:35.004 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:32:35.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574852 -lt 55834574851 2022-03-01T06:32:35.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2022-03-01T06:32:35.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:32:35.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:32:35.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:32:35.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:32:35.327 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2022-03-01T06:32:35.328 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:32:35.328 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:32:35.328 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:32:35.328 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:32:35.328 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:32:35.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:32:35.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:32:35.414 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:32:35.414 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:32:35.414 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:32:35.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:32:35.736 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:32:35.737 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:32:35.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:32:35.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:35.738 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:32:35.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2022-03-01T06:32:35.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2022-03-01T06:32:35.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2022-03-01T06:32:35.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:35.861 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:32:35.984 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705672 2022-03-01T06:32:35.984 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705672 2022-03-01T06:32:35.985 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-38654705672' 2022-03-01T06:32:35.985 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:35.985 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:32:36.107 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574854 2022-03-01T06:32:36.107 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574854 2022-03-01T06:32:36.108 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-55834574854' 2022-03-01T06:32:36.108 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:36.108 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2022-03-01T06:32:36.109 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:36.110 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:32:36.110 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2022-03-01T06:32:36.110 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:36.110 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2022-03-01T06:32:36.111 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2022-03-01T06:32:36.111 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836490 2022-03-01T06:32:36.112 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:32:36.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836490 2022-03-01T06:32:36.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:32:37.433 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:32:37.433 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:32:37.767 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2022-03-01T06:32:37.768 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:37.768 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705672 2022-03-01T06:32:37.768 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:37.769 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:32:37.769 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705672 2022-03-01T06:32:37.769 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:37.770 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705672 2022-03-01T06:32:37.770 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2022-03-01T06:32:37.771 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 38654705672 2022-03-01T06:32:37.771 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:32:38.093 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705672 -lt 38654705672 2022-03-01T06:32:38.094 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:38.094 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-55834574854 2022-03-01T06:32:38.095 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:38.095 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:32:38.096 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-55834574854 2022-03-01T06:32:38.096 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:38.097 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574854 2022-03-01T06:32:38.097 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2022-03-01T06:32:38.098 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 55834574854 2022-03-01T06:32:38.098 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:32:38.437 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574855 -lt 55834574854 2022-03-01T06:32:38.438 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:32:38.439 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:32:38.439 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:32:38.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:32:38.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:32:38.859 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:32:38.860 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:32:38.860 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:32:38.860 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:32:38.860 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:32:38.861 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:32:39.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-03-01T06:32:39.173 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:32:39.173 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:32:39.173 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:32:39.597 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-03-01T06:32:39.597 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-03-01T06:32:39.597 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-03-01T06:32:39.597 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2022-03-01T06:32:39.598 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2022-03-01T06:32:39.901 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:32:39.912 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2022-03-01T06:32:39.913 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2022-03-01T06:32:39.914 INFO:tasks.workunit.client.0.smithi169.stdout:primary and soon to be divergent is 1 2022-03-01T06:32:39.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2022-03-01T06:32:40.217 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:32:40.217 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-03-01T06:32:29.546123+0000 0'0 24:13 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:32:40.218 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:32:40.218 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:32:40.219 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:32:40.228 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2022-03-01T06:32:40.228 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2022-03-01T06:32:40.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2022-03-01T06:32:40.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2022-03-01T06:32:40.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2022-03-01T06:32:40.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2022-03-01T06:32:40.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2022-03-01T06:32:40.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2022-03-01T06:32:40.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2022-03-01T06:32:40.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2022-03-01T06:32:40.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2022-03-01T06:32:40.231 INFO:tasks.workunit.client.0.smithi169.stdout:writing initial objects 2022-03-01T06:32:40.232 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2022-03-01T06:32:40.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2022-03-01T06:32:40.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2022-03-01T06:32:40.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2022-03-01T06:32:40.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2022-03-01T06:32:40.667 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.667 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2022-03-01T06:32:40.784 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.784 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2022-03-01T06:32:40.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:40.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2022-03-01T06:32:41.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2022-03-01T06:32:41.111 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.111 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2022-03-01T06:32:41.262 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.262 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2022-03-01T06:32:41.404 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.404 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2022-03-01T06:32:41.572 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.572 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2022-03-01T06:32:41.689 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.689 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2022-03-01T06:32:41.790 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.790 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2022-03-01T06:32:41.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:41.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2022-03-01T06:32:42.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2022-03-01T06:32:42.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2022-03-01T06:32:42.451 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.451 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2022-03-01T06:32:42.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2022-03-01T06:32:42.686 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.686 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2022-03-01T06:32:42.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2022-03-01T06:32:42.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:42.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2022-03-01T06:32:43.013 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.013 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2022-03-01T06:32:43.114 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.114 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2022-03-01T06:32:43.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2022-03-01T06:32:43.423 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.424 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2022-03-01T06:32:43.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2022-03-01T06:32:43.650 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.650 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2022-03-01T06:32:43.759 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.759 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2022-03-01T06:32:43.892 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:43.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2022-03-01T06:32:44.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2022-03-01T06:32:44.110 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.111 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2022-03-01T06:32:44.262 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.262 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2022-03-01T06:32:44.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2022-03-01T06:32:44.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.513 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2022-03-01T06:32:44.622 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.623 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2022-03-01T06:32:44.731 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.731 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2022-03-01T06:32:44.831 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.832 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2022-03-01T06:32:44.924 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:44.924 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2022-03-01T06:32:45.041 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.041 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2022-03-01T06:32:45.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2022-03-01T06:32:45.292 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.293 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2022-03-01T06:32:45.402 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.402 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2022-03-01T06:32:45.544 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.544 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2022-03-01T06:32:45.653 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.653 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2022-03-01T06:32:45.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2022-03-01T06:32:45.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:45.905 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2022-03-01T06:32:46.013 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.013 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2022-03-01T06:32:46.115 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.115 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2022-03-01T06:32:46.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2022-03-01T06:32:46.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2022-03-01T06:32:46.566 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.566 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2022-03-01T06:32:46.675 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.675 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2022-03-01T06:32:46.784 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.785 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2022-03-01T06:32:46.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:46.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2022-03-01T06:32:47.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.003 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2022-03-01T06:32:47.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2022-03-01T06:32:47.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2022-03-01T06:32:47.345 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.346 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2022-03-01T06:32:47.497 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.497 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2022-03-01T06:32:47.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.723 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2022-03-01T06:32:47.839 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.840 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2022-03-01T06:32:47.940 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:47.941 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2022-03-01T06:32:48.041 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.041 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2022-03-01T06:32:48.150 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.150 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2022-03-01T06:32:48.267 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.268 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2022-03-01T06:32:48.393 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.393 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2022-03-01T06:32:48.502 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.502 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2022-03-01T06:32:48.611 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.611 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2022-03-01T06:32:48.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2022-03-01T06:32:48.829 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.829 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2022-03-01T06:32:48.971 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:48.971 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2022-03-01T06:32:49.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2022-03-01T06:32:49.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2022-03-01T06:32:49.315 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.315 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2022-03-01T06:32:49.507 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.508 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2022-03-01T06:32:49.616 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.617 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2022-03-01T06:32:49.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2022-03-01T06:32:49.833 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.834 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2022-03-01T06:32:49.942 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:49.943 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2022-03-01T06:32:50.051 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.052 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2022-03-01T06:32:50.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2022-03-01T06:32:50.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2022-03-01T06:32:50.428 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.428 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2022-03-01T06:32:50.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2022-03-01T06:32:50.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2022-03-01T06:32:50.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2022-03-01T06:32:50.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2022-03-01T06:32:50.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:50.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2022-03-01T06:32:51.099 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.099 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2022-03-01T06:32:51.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2022-03-01T06:32:51.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2022-03-01T06:32:51.484 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.485 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2022-03-01T06:32:51.593 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.594 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2022-03-01T06:32:51.702 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.703 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2022-03-01T06:32:51.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2022-03-01T06:32:51.929 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:51.929 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2022-03-01T06:32:52.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:52.047 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2022-03-01T06:32:52.163 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:52.163 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2022-03-01T06:32:52.307 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2022-03-01T06:32:52.307 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2022-03-01T06:32:52.457 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:32:52.458 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2022-03-01T06:32:52.458 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:32:52.458 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:32:52.458 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:32:52.459 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:32:52.459 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:32:52.459 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:32:52.459 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:32:52.459 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:32:52.460 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:32:52.460 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:32:52.511 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:32:52.511 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:32:52.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:32:52.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:32:52.512 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:32:52.831 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:32:52.831 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:32:52.832 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:32:52.832 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:32:52.832 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:52.832 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:32:52.955 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2022-03-01T06:32:52.955 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2022-03-01T06:32:52.956 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2022-03-01T06:32:52.956 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:52.956 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:32:53.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705677 2022-03-01T06:32:53.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705677 2022-03-01T06:32:53.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-38654705677' 2022-03-01T06:32:53.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:32:53.080 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:32:53.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574860 2022-03-01T06:32:53.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574860 2022-03-01T06:32:53.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-38654705677 2-55834574860' 2022-03-01T06:32:53.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:53.204 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2022-03-01T06:32:53.204 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:53.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:32:53.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2022-03-01T06:32:53.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:53.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2022-03-01T06:32:53.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2022-03-01T06:32:53.207 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836495 2022-03-01T06:32:53.208 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:32:53.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2022-03-01T06:32:53.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:53.530 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705677 2022-03-01T06:32:53.530 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:53.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:32:53.531 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705677 2022-03-01T06:32:53.531 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:53.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705677 2022-03-01T06:32:53.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2022-03-01T06:32:53.533 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 38654705677 2022-03-01T06:32:53.533 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:32:53.858 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705677 -lt 38654705677 2022-03-01T06:32:53.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:32:53.859 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-55834574860 2022-03-01T06:32:53.859 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:32:53.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:32:53.860 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-55834574860 2022-03-01T06:32:53.860 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:32:53.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574860 2022-03-01T06:32:53.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2022-03-01T06:32:53.862 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 55834574860 2022-03-01T06:32:53.862 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:32:54.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574860 -lt 55834574860 2022-03-01T06:32:54.184 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:32:54.184 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:32:54.185 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:32:54.622 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:32:54.622 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:32:54.622 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:32:54.623 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:32:54.623 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:32:54.623 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:32:54.623 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:32:54.624 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:32:54.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-03-01T06:32:54.938 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:32:54.938 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:32:54.939 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:32:55.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-03-01T06:32:55.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-03-01T06:32:55.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-03-01T06:32:55.360 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2022-03-01T06:32:55.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2022-03-01T06:32:55.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2022-03-01T06:32:55.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2022-03-01T06:32:55.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2022-03-01T06:32:55.689 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2022-03-01T06:32:55.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2022-03-01T06:32:55.690 INFO:tasks.workunit.client.0.smithi169.stdout:blackholing osds 0 2 2022-03-01T06:32:55.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2022-03-01T06:32:55.992 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:32:55.992 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2022-03-01T06:32:29.546123+0000 26'100 26:113 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:32:55.993 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:32:55.993 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:32:55.994 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:32:56.002 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2022-03-01T06:32:56.003 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2022-03-01T06:32:56.004 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-03-01T06:32:56.004 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-03-01T06:32:56.004 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:32:56.004 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:56.004 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:56.005 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.0.asok 2022-03-01T06:32:56.005 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2022-03-01T06:32:56.005 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.0.asok config set objectstore_blackhole 1 2022-03-01T06:32:56.084 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:32:56.085 INFO:tasks.workunit.client.0.smithi169.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-03-01T06:32:56.085 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:32:56.091 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2022-03-01T06:32:56.093 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2022-03-01T06:32:56.093 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-03-01T06:32:56.093 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-03-01T06:32:56.094 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:32:56.094 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:56.094 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:56.095 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.2.asok 2022-03-01T06:32:56.095 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2022-03-01T06:32:56.095 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.2.asok config set objectstore_blackhole 1 2022-03-01T06:32:56.175 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:32:56.176 INFO:tasks.workunit.client.0.smithi169.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-03-01T06:32:56.176 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:32:56.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2022-03-01T06:32:56.184 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2022-03-01T06:32:56.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2022-03-01T06:32:56.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2022-03-01T06:32:56.185 INFO:tasks.workunit.client.0.smithi169.stdout:writing divergent object 2022-03-01T06:32:56.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2022-03-01T06:32:56.186 INFO:tasks.workunit.client.0.smithi169.stdout:create missing divergent object 2022-03-01T06:32:56.188 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2022-03-01T06:32:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2022-03-01T06:32:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2022-03-01T06:32:56.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2022-03-01T06:32:56.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2022-03-01T06:32:56.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2022-03-01T06:32:56.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test 2022-03-01T06:32:56.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2022-03-01T06:32:56.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=existing_99 2022-03-01T06:32:56.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2022-03-01T06:32:56.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/divergent-priors 2022-03-01T06:32:56.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2022-03-01T06:32:56.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2022-03-01T06:32:56.192 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2022-03-01T06:32:56.192 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2022-03-01T06:32:56.192 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test existing_99 2022-03-01T06:32:56.193 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2022-03-01T06:32:56.193 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=existing_99 2022-03-01T06:32:56.193 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test existing_99 2022-03-01T06:32:56.194 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2022-03-01T06:32:56.516 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2022-03-01T06:32:56.516 INFO:tasks.workunit.client.0.smithi169.stderr:0 2022-03-01T06:32:56.516 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:32:56.517 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2022-03-01T06:32:56.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2022-03-01T06:32:56.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2022-03-01T06:32:56.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2022-03-01T06:32:56.518 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2022-03-01T06:32:56.518 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/divergent-priors/1/type 2022-03-01T06:32:56.520 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2022-03-01T06:32:56.520 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2022-03-01T06:32:56.520 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2022-03-01T06:32:56.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2022-03-01T06:32:56.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2022-03-01T06:32:56.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2022-03-01T06:32:56.522 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2022-03-01T06:32:56.523 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2022-03-01T06:32:56.523 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-03-01T06:32:56.523 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-03-01T06:32:56.523 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:32:56.523 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:56.524 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:56.524 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.1.asok 2022-03-01T06:32:56.524 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.31520/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2022-03-01T06:32:56.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2022-03-01T06:32:56.610 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2022-03-01T06:32:56.610 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2022-03-01T06:32:56.610 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2022-03-01T06:32:56.610 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-03-01T06:32:56.611 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-03-01T06:32:56.611 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:32:56.611 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:32:56.611 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:32:56.612 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.1.asok 2022-03-01T06:32:56.612 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2022-03-01T06:32:56.612 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.1.asok injectdataerr test existing_99 2022-03-01T06:32:56.701 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2022-03-01T06:32:56.701 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2022-03-01T06:33:06.702 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2022-03-01T06:33:06.715 INFO:tasks.workunit.client.0.smithi169.stderr:rados: no process found 2022-03-01T06:33:06.716 INFO:tasks.workunit.client.0.smithi169.stdout:killing all the osds 2022-03-01T06:33:06.716 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2022-03-01T06:33:06.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:07.027 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:07.027 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2022-03-01T06:32:57.079930+0000 26'101 26:122 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:07.027 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:07.028 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:07.031 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:07.037 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2022-03-01T06:33:07.038 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:33:07.038 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:33:07.039 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:33:07.039 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:33:07.039 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:33:07.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:33:07.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2022-03-01T06:33:07.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2022-03-01T06:33:07.655 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already down. 2022-03-01T06:33:07.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2022-03-01T06:33:07.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2022-03-01T06:33:08.189 INFO:tasks.workunit.client.0.smithi169.stderr:osd.1 is already down. 2022-03-01T06:33:08.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2022-03-01T06:33:08.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2022-03-01T06:33:08.707 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already down. 2022-03-01T06:33:08.715 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:08.716 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2022-03-01T06:33:09.181 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already out. 2022-03-01T06:33:09.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:09.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2022-03-01T06:33:09.666 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already out. 2022-03-01T06:33:09.674 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2022-03-01T06:33:09.675 INFO:tasks.workunit.client.0.smithi169.stdout:bringing up non_divergent 0 2 2022-03-01T06:33:09.675 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:09.976 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:09.977 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2022-03-01T06:32:57.079930+0000 26'101 26:122 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:09.977 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:09.977 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:09.978 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:09.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:09.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2022-03-01T06:33:09.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:33:09.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:33:09.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2022-03-01T06:33:09.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:33:09.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2022-03-01T06:33:09.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:33:09.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:33:09.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:33:09.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:33:09.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-03-01T06:33:09.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-03-01T06:33:09.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:33:09.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:33:09.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:33:09.994 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:33:09.994 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:33:09.994 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:33:09.994 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:33:09.995 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:33:09.995 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:33:09.995 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:33:09.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:33:09.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:33:09.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:33:09.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:33:09.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:33:09.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:33:09.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:33:09.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:33:09.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:33:09.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2022-03-01T06:33:09.998 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2022-03-01T06:33:09.998 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:33:09.999 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:33:09.999 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2022-03-01T06:33:09.999 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2022-03-01T06:33:10.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:33:10.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:33:10.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:33:10.016 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:10.035+0000 7f560f027f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:10.018 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:10.037+0000 7f560f027f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:10.019 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:10.038+0000 7f560f027f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:10.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2022-03-01T06:33:10.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:33:10.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:33:10.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:33:10.328 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:33:10.328 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:10.328 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:33:10.328 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:33:10.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:10.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:33:10.537 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:10.557+0000 7f560f027f00 -1 Falling back to public interface 2022-03-01T06:33:10.654 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:33:11.459 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:11.478+0000 7f560f027f00 -1 osd.0 26 log_to_monitors {default=true} 2022-03-01T06:33:11.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:33:11.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:11.656 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:33:11.657 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:33:11.657 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:11.657 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:33:11.985 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up out weight 0 up_from 33 up_thru 0 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/36648,v1:127.0.0.1:6803/36648] [v2:127.0.0.1:6804/36648,v1:127.0.0.1:6805/36648] exists,up b36cc591-5c05-4344-a0c6-fd7c20be6391 2022-03-01T06:33:11.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:33:11.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:33:11.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:33:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2022-03-01T06:33:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:33:11.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:33:11.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2022-03-01T06:33:11.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:33:11.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2022-03-01T06:33:11.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:33:11.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:33:11.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:33:11.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:33:11.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-03-01T06:33:11.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-03-01T06:33:11.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:33:11.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:33:11.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:33:11.994 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:33:11.994 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:33:11.995 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:33:11.995 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:33:11.995 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:33:11.995 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:33:11.995 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:33:11.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:33:11.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:33:11.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:33:11.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:33:11.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:33:11.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:33:11.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:33:11.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:33:11.998 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:33:11.998 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2022-03-01T06:33:11.998 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2022-03-01T06:33:11.999 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:33:12.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:33:12.000 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2022-03-01T06:33:12.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2022-03-01T06:33:12.001 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:33:12.001 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:33:12.001 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:33:12.014 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:12.034+0000 7fe641e4cf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:12.016 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:12.036+0000 7fe641e4cf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:12.017 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:12.037+0000 7fe641e4cf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:12.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2022-03-01T06:33:12.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:33:12.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:33:12.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:33:12.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:33:12.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:12.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:33:12.333 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:33:12.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:12.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:33:12.541 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:12.561+0000 7fe641e4cf00 -1 Falling back to public interface 2022-03-01T06:33:12.663 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:33:13.618 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:13.637+0000 7fe641e4cf00 -1 osd.2 26 log_to_monitors {default=true} 2022-03-01T06:33:13.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:33:13.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:13.665 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:33:13.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:33:13.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:13.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:33:14.056 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [13,26) [v2:127.0.0.1:6810/36842,v1:127.0.0.1:6811/36842] [v2:127.0.0.1:6812/36842,v1:127.0.0.1:6813/36842] exists,up 9d15391e-aa51-4a5b-868e-961f98e20902 2022-03-01T06:33:14.058 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:33:14.058 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:33:14.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:33:14.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:14.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2022-03-01T06:33:14.526 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already in. 2022-03-01T06:33:14.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:14.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2022-03-01T06:33:15.018 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already in. 2022-03-01T06:33:15.027 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:33:15.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2022-03-01T06:33:15.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:33:15.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:33:15.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:33:15.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:33:15.029 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:33:15.029 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:33:15.029 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:33:15.030 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:33:15.030 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:33:15.030 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:33:15.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:33:15.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:33:15.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:33:15.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:33:15.083 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:33:15.421 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:33:15.422 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:33:15.422 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:33:15.422 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:33:15.422 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:33:15.423 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:33:15.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920771 2022-03-01T06:33:15.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920771 2022-03-01T06:33:15.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-141733920771' 2022-03-01T06:33:15.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:33:15.549 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:33:15.636 INFO:tasks.workunit.client.0.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-03-01T06:33:15.638 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-03-01T06:33:15.638 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-03-01T06:33:15.638 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-03-01T06:33:15.639 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:33:15.639 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:33:15.762 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855363 2022-03-01T06:33:15.762 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855363 2022-03-01T06:33:15.763 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-141733920771 2-150323855363' 2022-03-01T06:33:15.763 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:33:15.763 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-141733920771 2022-03-01T06:33:15.763 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:33:15.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:33:15.764 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-141733920771 2022-03-01T06:33:15.765 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:33:15.765 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920771 2022-03-01T06:33:15.765 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2022-03-01T06:33:15.766 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 141733920771 2022-03-01T06:33:15.766 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:33:16.085 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 141733920771 2022-03-01T06:33:16.085 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:33:17.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:33:17.087 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:33:17.424 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 141733920771 2022-03-01T06:33:17.424 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:33:18.422 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 299 -eq 0 ']' 2022-03-01T06:33:18.422 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:33:18.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920772 -lt 141733920771 2022-03-01T06:33:18.748 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:33:18.748 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-150323855363 2022-03-01T06:33:18.748 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:33:18.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:33:18.749 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-150323855363 2022-03-01T06:33:18.749 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:33:18.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855363 2022-03-01T06:33:18.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2022-03-01T06:33:18.751 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 150323855363 2022-03-01T06:33:18.751 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:33:19.074 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855363 -lt 150323855363 2022-03-01T06:33:19.075 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:33:19.075 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:19.075 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:19.494 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:33:19.495 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:19.495 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:19.495 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:19.495 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:19.496 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:19.496 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:19.496 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:19.815 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:19.816 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:19.816 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:19.816 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:20.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:20.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2022-03-01T06:33:20.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2022-03-01T06:33:20.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2022-03-01T06:33:20.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:20.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2022-03-01T06:33:20.337 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:20.337 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:20.338 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:20.338 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:20.338 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:20.338 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:20.339 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:20.339 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:20.652 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:20.652 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:20.653 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:20.653 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:21.068 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:21.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:21.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:21.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:21.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:21.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:21.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:21.070 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:21.070 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:21.489 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:21.490 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:21.490 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2022-03-01T06:33:21.490 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:21.491 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2022-03-01T06:33:21.691 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:21.691 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:21.692 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:21.692 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:21.692 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:21.693 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:21.693 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:21.693 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:22.005 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:22.006 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:22.006 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:22.006 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:22.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:22.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:22.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:22.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:22.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:22.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:22.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:22.433 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:22.433 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:22.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:22.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:22.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2022-03-01T06:33:22.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:22.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2022-03-01T06:33:23.252 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:23.252 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:23.252 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:23.253 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:23.253 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:23.253 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:23.254 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:23.254 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:23.564 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:23.565 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:23.565 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:23.565 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:23.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:23.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:23.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:23.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:23.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:23.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:23.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:23.993 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:23.993 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:24.409 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:24.409 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:24.410 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2022-03-01T06:33:24.410 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:24.411 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2022-03-01T06:33:25.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:25.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:25.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:25.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:25.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:25.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:25.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:25.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:25.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:25.595 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:25.595 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:25.595 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:25.953 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:25.954 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:25.954 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:25.954 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:25.955 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:25.955 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:25.955 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:25.956 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:25.956 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:26.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:26.372 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:26.372 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2022-03-01T06:33:26.372 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:26.373 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2022-03-01T06:33:27.972 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:27.972 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:27.973 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:27.974 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:27.974 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:27.974 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:27.974 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:27.975 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:28.297 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:28.298 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:28.298 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:28.298 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:28.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:28.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:28.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:28.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:28.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:28.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:28.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:28.721 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:28.722 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:29.140 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:29.140 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:29.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2022-03-01T06:33:29.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:29.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2022-03-01T06:33:32.340 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:32.341 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:32.342 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:32.342 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:32.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:32.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:32.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:32.343 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:32.694 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:32.694 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:32.695 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:32.695 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:33.114 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:33.114 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:33.115 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:33.115 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:33.115 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:33.115 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:33.116 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:33.116 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:33.116 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:33.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:33.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:33.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2022-03-01T06:33:33.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:33.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2022-03-01T06:33:39.938 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:39.938 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:39.939 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:39.939 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:39.939 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:39.939 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:39.939 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:39.940 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:40.266 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:40.267 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:40.267 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:40.267 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:40.684 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:40.685 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:40.685 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:40.685 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:40.686 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:40.686 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:40.686 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:40.687 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:40.687 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:41.105 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:41.106 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:41.106 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2022-03-01T06:33:41.107 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:33:41.107 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2022-03-01T06:33:48.406 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:33:48.407 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:33:48.407 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:33:48.407 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:33:48.408 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:33:48.408 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:33:48.408 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:33:48.408 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:33:48.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:33:48.732 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:33:48.732 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:33:48.733 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:33:49.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:33:49.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:33:49.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:33:49.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:33:49.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:33:49.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:33:49.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:33:49.149 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:33:49.149 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:33:49.598 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:33:49.598 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:33:49.598 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2022-03-01T06:33:49.599 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2022-03-01T06:33:50.012 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:33:50.012 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_fingerprint": "09bf7a72-284a-4220-935f-6af3aa2acfca", 2022-03-01T06:33:50.012 INFO:tasks.workunit.client.0.smithi169.stdout: "version": "15.2.15-249-g45ece157", 2022-03-01T06:33:50.013 INFO:tasks.workunit.client.0.smithi169.stdout: "commit": "45ece157fc29b55a5ee952c81c0c23abdb89bf98", 2022-03-01T06:33:50.013 INFO:tasks.workunit.client.0.smithi169.stdout: "timestamp": "2022-03-01T06:33:50.031357+0000", 2022-03-01T06:33:50.013 INFO:tasks.workunit.client.0.smithi169.stdout: "tag": "", 2022-03-01T06:33:50.013 INFO:tasks.workunit.client.0.smithi169.stdout: "health": { 2022-03-01T06:33:50.014 INFO:tasks.workunit.client.0.smithi169.stdout: "status": "HEALTH_WARN", 2022-03-01T06:33:50.014 INFO:tasks.workunit.client.0.smithi169.stdout: "checks": { 2022-03-01T06:33:50.014 INFO:tasks.workunit.client.0.smithi169.stdout: "OSDMAP_FLAGS": { 2022-03-01T06:33:50.014 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:33:50.015 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:33:50.015 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "nodown,noout,noin flag(s) set", 2022-03-01T06:33:50.015 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 17 2022-03-01T06:33:50.015 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.016 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [], 2022-03-01T06:33:50.016 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:33:50.016 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.016 INFO:tasks.workunit.client.0.smithi169.stdout: "OSD_DOWN": { 2022-03-01T06:33:50.017 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:33:50.017 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:33:50.017 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "1 osds down", 2022-03-01T06:33:50.017 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:33:50.018 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.018 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:33:50.018 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.018 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "osd.1 (root=default,host=smithi169) is down" 2022-03-01T06:33:50.018 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.019 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.019 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:33:50.019 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.019 INFO:tasks.workunit.client.0.smithi169.stdout: "PG_DEGRADED": { 2022-03-01T06:33:50.020 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:33:50.020 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:33:50.020 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2022-03-01T06:33:50.021 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:33:50.021 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.021 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:33:50.021 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.022 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2022-03-01T06:33:50.022 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.022 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.023 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:33:50.023 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.023 INFO:tasks.workunit.client.0.smithi169.stdout: "POOL_APP_NOT_ENABLED": { 2022-03-01T06:33:50.024 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:33:50.024 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:33:50.024 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "1 pool(s) do not have an application enabled", 2022-03-01T06:33:50.024 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:33:50.025 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.025 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:33:50.025 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.025 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "application not enabled on pool 'test'" 2022-03-01T06:33:50.025 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.026 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.026 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2022-03-01T06:33:50.027 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.027 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.027 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:33:50.027 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.028 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.028 INFO:tasks.workunit.client.0.smithi169.stdout: "mutes": [] 2022-03-01T06:33:50.028 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.028 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap_first_committed": 1, 2022-03-01T06:33:50.029 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap_last_committed": 1, 2022-03-01T06:33:50.029 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap": { 2022-03-01T06:33:50.029 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 1, 2022-03-01T06:33:50.029 INFO:tasks.workunit.client.0.smithi169.stdout: "fsid": "03da9435-ccbd-45e1-be8b-88e9f3ce57a1", 2022-03-01T06:33:50.030 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:32:06.584697Z", 2022-03-01T06:33:50.030 INFO:tasks.workunit.client.0.smithi169.stdout: "created": "2022-03-01T06:32:06.584697Z", 2022-03-01T06:33:50.030 INFO:tasks.workunit.client.0.smithi169.stdout: "min_mon_release": 15, 2022-03-01T06:33:50.030 INFO:tasks.workunit.client.0.smithi169.stdout: "min_mon_release_name": "octopus", 2022-03-01T06:33:50.031 INFO:tasks.workunit.client.0.smithi169.stdout: "features": { 2022-03-01T06:33:50.031 INFO:tasks.workunit.client.0.smithi169.stdout: "persistent": [ 2022-03-01T06:33:50.031 INFO:tasks.workunit.client.0.smithi169.stdout: "kraken", 2022-03-01T06:33:50.031 INFO:tasks.workunit.client.0.smithi169.stdout: "luminous", 2022-03-01T06:33:50.032 INFO:tasks.workunit.client.0.smithi169.stdout: "mimic", 2022-03-01T06:33:50.032 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap-prune", 2022-03-01T06:33:50.032 INFO:tasks.workunit.client.0.smithi169.stdout: "nautilus", 2022-03-01T06:33:50.032 INFO:tasks.workunit.client.0.smithi169.stdout: "octopus" 2022-03-01T06:33:50.032 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.033 INFO:tasks.workunit.client.0.smithi169.stdout: "optional": [] 2022-03-01T06:33:50.033 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.033 INFO:tasks.workunit.client.0.smithi169.stdout: "mons": [ 2022-03-01T06:33:50.033 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.034 INFO:tasks.workunit.client.0.smithi169.stdout: "rank": 0, 2022-03-01T06:33:50.034 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "a", 2022-03-01T06:33:50.034 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:33:50.034 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.034 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.035 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.035 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:7115", 2022-03-01T06:33:50.035 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 0 2022-03-01T06:33:50.035 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.036 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.036 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.036 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:7115/0", 2022-03-01T06:33:50.036 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:7115/0", 2022-03-01T06:33:50.037 INFO:tasks.workunit.client.0.smithi169.stdout: "priority": 0, 2022-03-01T06:33:50.037 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 0 2022-03-01T06:33:50.037 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.037 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.037 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.038 INFO:tasks.workunit.client.0.smithi169.stdout: "quorum": [ 2022-03-01T06:33:50.038 INFO:tasks.workunit.client.0.smithi169.stdout: 0 2022-03-01T06:33:50.038 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.039 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap": { 2022-03-01T06:33:50.039 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 38, 2022-03-01T06:33:50.039 INFO:tasks.workunit.client.0.smithi169.stdout: "fsid": "03da9435-ccbd-45e1-be8b-88e9f3ce57a1", 2022-03-01T06:33:50.039 INFO:tasks.workunit.client.0.smithi169.stdout: "created": "2022-03-01T06:32:07.324816+0000", 2022-03-01T06:33:50.039 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:33:15.188050+0000", 2022-03-01T06:33:50.040 INFO:tasks.workunit.client.0.smithi169.stdout: "last_up_change": "2022-03-01T06:33:13.876183+0000", 2022-03-01T06:33:50.040 INFO:tasks.workunit.client.0.smithi169.stdout: "last_in_change": "2022-03-01T06:33:14.860941+0000", 2022-03-01T06:33:50.040 INFO:tasks.workunit.client.0.smithi169.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-03-01T06:33:50.040 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_num": 5800384, 2022-03-01T06:33:50.041 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_set": [ 2022-03-01T06:33:50.041 INFO:tasks.workunit.client.0.smithi169.stdout: "nodown", 2022-03-01T06:33:50.041 INFO:tasks.workunit.client.0.smithi169.stdout: "noin", 2022-03-01T06:33:50.041 INFO:tasks.workunit.client.0.smithi169.stdout: "noout", 2022-03-01T06:33:50.042 INFO:tasks.workunit.client.0.smithi169.stdout: "pglog_hardlimit", 2022-03-01T06:33:50.042 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snapdirs", 2022-03-01T06:33:50.042 INFO:tasks.workunit.client.0.smithi169.stdout: "recovery_deletes", 2022-03-01T06:33:50.042 INFO:tasks.workunit.client.0.smithi169.stdout: "sortbitwise" 2022-03-01T06:33:50.043 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.043 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_version": 9, 2022-03-01T06:33:50.044 INFO:tasks.workunit.client.0.smithi169.stdout: "full_ratio": 0.99000000953674316, 2022-03-01T06:33:50.044 INFO:tasks.workunit.client.0.smithi169.stdout: "backfillfull_ratio": 0.99000000953674316, 2022-03-01T06:33:50.044 INFO:tasks.workunit.client.0.smithi169.stdout: "nearfull_ratio": 0.85000002384185791, 2022-03-01T06:33:50.044 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_snapshot": "", 2022-03-01T06:33:50.045 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_max": 1, 2022-03-01T06:33:50.045 INFO:tasks.workunit.client.0.smithi169.stdout: "max_osd": 3, 2022-03-01T06:33:50.045 INFO:tasks.workunit.client.0.smithi169.stdout: "require_min_compat_client": "jewel", 2022-03-01T06:33:50.045 INFO:tasks.workunit.client.0.smithi169.stdout: "min_compat_client": "jewel", 2022-03-01T06:33:50.045 INFO:tasks.workunit.client.0.smithi169.stdout: "require_osd_release": "octopus", 2022-03-01T06:33:50.046 INFO:tasks.workunit.client.0.smithi169.stdout: "pools": [ 2022-03-01T06:33:50.046 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.046 INFO:tasks.workunit.client.0.smithi169.stdout: "pool": 1, 2022-03-01T06:33:50.047 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_name": "test", 2022-03-01T06:33:50.047 INFO:tasks.workunit.client.0.smithi169.stdout: "create_time": "2022-03-01T06:32:28.518339+0000", 2022-03-01T06:33:50.047 INFO:tasks.workunit.client.0.smithi169.stdout: "flags": 1, 2022-03-01T06:33:50.048 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_names": "hashpspool", 2022-03-01T06:33:50.048 INFO:tasks.workunit.client.0.smithi169.stdout: "type": 1, 2022-03-01T06:33:50.048 INFO:tasks.workunit.client.0.smithi169.stdout: "size": 3, 2022-03-01T06:33:50.048 INFO:tasks.workunit.client.0.smithi169.stdout: "min_size": 2, 2022-03-01T06:33:50.049 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_rule": 0, 2022-03-01T06:33:50.049 INFO:tasks.workunit.client.0.smithi169.stdout: "object_hash": 2, 2022-03-01T06:33:50.049 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_autoscale_mode": "off", 2022-03-01T06:33:50.049 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num": 1, 2022-03-01T06:33:50.050 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_placement_num": 1, 2022-03-01T06:33:50.050 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_placement_num_target": 1, 2022-03-01T06:33:50.050 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num_target": 1, 2022-03-01T06:33:50.050 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num_pending": 1, 2022-03-01T06:33:50.051 INFO:tasks.workunit.client.0.smithi169.stdout: "last_pg_merge_meta": { 2022-03-01T06:33:50.051 INFO:tasks.workunit.client.0.smithi169.stdout: "source_pgid": "0.0", 2022-03-01T06:33:50.051 INFO:tasks.workunit.client.0.smithi169.stdout: "ready_epoch": 0, 2022-03-01T06:33:50.052 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_started": 0, 2022-03-01T06:33:50.052 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_clean": 0, 2022-03-01T06:33:50.052 INFO:tasks.workunit.client.0.smithi169.stdout: "source_version": "0'0", 2022-03-01T06:33:50.052 INFO:tasks.workunit.client.0.smithi169.stdout: "target_version": "0'0" 2022-03-01T06:33:50.053 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.053 INFO:tasks.workunit.client.0.smithi169.stdout: "last_change": "26", 2022-03-01T06:33:50.053 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend": "0", 2022-03-01T06:33:50.054 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2022-03-01T06:33:50.054 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2022-03-01T06:33:50.054 INFO:tasks.workunit.client.0.smithi169.stdout: "auid": 0, 2022-03-01T06:33:50.054 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_mode": "selfmanaged", 2022-03-01T06:33:50.055 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_seq": 0, 2022-03-01T06:33:50.055 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_epoch": 0, 2022-03-01T06:33:50.055 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_snaps": [], 2022-03-01T06:33:50.055 INFO:tasks.workunit.client.0.smithi169.stdout: "removed_snaps": "[]", 2022-03-01T06:33:50.056 INFO:tasks.workunit.client.0.smithi169.stdout: "quota_max_bytes": 0, 2022-03-01T06:33:50.056 INFO:tasks.workunit.client.0.smithi169.stdout: "quota_max_objects": 0, 2022-03-01T06:33:50.056 INFO:tasks.workunit.client.0.smithi169.stdout: "tiers": [], 2022-03-01T06:33:50.056 INFO:tasks.workunit.client.0.smithi169.stdout: "tier_of": -1, 2022-03-01T06:33:50.057 INFO:tasks.workunit.client.0.smithi169.stdout: "read_tier": -1, 2022-03-01T06:33:50.057 INFO:tasks.workunit.client.0.smithi169.stdout: "write_tier": -1, 2022-03-01T06:33:50.058 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_mode": "none", 2022-03-01T06:33:50.058 INFO:tasks.workunit.client.0.smithi169.stdout: "target_max_bytes": 0, 2022-03-01T06:33:50.059 INFO:tasks.workunit.client.0.smithi169.stdout: "target_max_objects": 0, 2022-03-01T06:33:50.059 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-03-01T06:33:50.059 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-03-01T06:33:50.060 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2022-03-01T06:33:50.060 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_min_flush_age": 0, 2022-03-01T06:33:50.061 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_min_evict_age": 0, 2022-03-01T06:33:50.061 INFO:tasks.workunit.client.0.smithi169.stdout: "erasure_code_profile": "", 2022-03-01T06:33:50.062 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_params": { 2022-03-01T06:33:50.062 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "none" 2022-03-01T06:33:50.062 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.063 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_period": 0, 2022-03-01T06:33:50.063 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_count": 0, 2022-03-01T06:33:50.064 INFO:tasks.workunit.client.0.smithi169.stdout: "use_gmt_hitset": true, 2022-03-01T06:33:50.064 INFO:tasks.workunit.client.0.smithi169.stdout: "min_read_recency_for_promote": 0, 2022-03-01T06:33:50.065 INFO:tasks.workunit.client.0.smithi169.stdout: "min_write_recency_for_promote": 0, 2022-03-01T06:33:50.065 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2022-03-01T06:33:50.065 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_search_last_n": 0, 2022-03-01T06:33:50.066 INFO:tasks.workunit.client.0.smithi169.stdout: "grade_table": [], 2022-03-01T06:33:50.066 INFO:tasks.workunit.client.0.smithi169.stdout: "stripe_width": 0, 2022-03-01T06:33:50.067 INFO:tasks.workunit.client.0.smithi169.stdout: "expected_num_objects": 0, 2022-03-01T06:33:50.067 INFO:tasks.workunit.client.0.smithi169.stdout: "fast_read": false, 2022-03-01T06:33:50.067 INFO:tasks.workunit.client.0.smithi169.stdout: "options": {}, 2022-03-01T06:33:50.068 INFO:tasks.workunit.client.0.smithi169.stdout: "application_metadata": {} 2022-03-01T06:33:50.068 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.069 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.069 INFO:tasks.workunit.client.0.smithi169.stdout: "osds": [ 2022-03-01T06:33:50.070 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.070 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:33:50.070 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "b36cc591-5c05-4344-a0c6-fd7c20be6391", 2022-03-01T06:33:50.071 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 1, 2022-03-01T06:33:50.071 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:33:50.072 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:33:50.072 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:33:50.073 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 5, 2022-03-01T06:33:50.073 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 26, 2022-03-01T06:33:50.074 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 33, 2022-03-01T06:33:50.074 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 37, 2022-03-01T06:33:50.074 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 27, 2022-03-01T06:33:50.075 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:33:50.075 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:33:50.076 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.076 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.077 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.077 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6802", 2022-03-01T06:33:50.077 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.078 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.078 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.079 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.079 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6803", 2022-03-01T06:33:50.080 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.080 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.080 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.081 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.081 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:33:50.082 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.082 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.082 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.083 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6804", 2022-03-01T06:33:50.083 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.083 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.084 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.084 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.084 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6805", 2022-03-01T06:33:50.085 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.085 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.086 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.086 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.086 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:33:50.087 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.087 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.088 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.088 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6808", 2022-03-01T06:33:50.088 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.089 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.089 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.090 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.090 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6809", 2022-03-01T06:33:50.090 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.090 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.091 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.091 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.091 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:33:50.091 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.091 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.092 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.092 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6806", 2022-03-01T06:33:50.092 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.092 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.093 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.093 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.093 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6807", 2022-03-01T06:33:50.093 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36648 2022-03-01T06:33:50.094 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.094 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.094 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.094 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6803/36648", 2022-03-01T06:33:50.095 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6805/36648", 2022-03-01T06:33:50.095 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6809/36648", 2022-03-01T06:33:50.095 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6807/36648", 2022-03-01T06:33:50.095 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:33:50.096 INFO:tasks.workunit.client.0.smithi169.stdout: "exists", 2022-03-01T06:33:50.096 INFO:tasks.workunit.client.0.smithi169.stdout: "up" 2022-03-01T06:33:50.096 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.096 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.097 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.097 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:33:50.097 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "fd55fbe2-fa36-4813-92e3-6da6f78c70a0", 2022-03-01T06:33:50.097 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 0, 2022-03-01T06:33:50.098 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:33:50.098 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:33:50.098 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:33:50.098 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 0, 2022-03-01T06:33:50.099 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 0, 2022-03-01T06:33:50.099 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 9, 2022-03-01T06:33:50.100 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 20, 2022-03-01T06:33:50.100 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 28, 2022-03-01T06:33:50.100 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:33:50.100 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:33:50.101 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.101 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.101 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.101 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6810", 2022-03-01T06:33:50.102 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.102 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.102 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.102 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.103 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6811", 2022-03-01T06:33:50.103 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.103 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.104 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.104 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.104 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:33:50.104 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.105 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.105 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.105 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6812", 2022-03-01T06:33:50.105 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.105 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.106 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.106 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.106 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6813", 2022-03-01T06:33:50.106 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.107 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.107 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.107 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.108 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:33:50.108 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.108 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.108 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.109 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6816", 2022-03-01T06:33:50.109 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.109 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.110 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.110 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.110 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6817", 2022-03-01T06:33:50.111 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.111 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.112 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.112 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.113 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:33:50.113 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.113 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.114 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.114 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6814", 2022-03-01T06:33:50.115 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.115 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.116 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.116 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.117 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6815", 2022-03-01T06:33:50.117 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 32308 2022-03-01T06:33:50.117 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.118 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.118 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.119 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6811/32308", 2022-03-01T06:33:50.119 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6813/32308", 2022-03-01T06:33:50.119 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6817/32308", 2022-03-01T06:33:50.120 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6815/32308", 2022-03-01T06:33:50.120 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:33:50.120 INFO:tasks.workunit.client.0.smithi169.stdout: "exists" 2022-03-01T06:33:50.121 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.121 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.121 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.121 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:33:50.122 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "9d15391e-aa51-4a5b-868e-961f98e20902", 2022-03-01T06:33:50.122 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 1, 2022-03-01T06:33:50.122 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:33:50.122 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:33:50.123 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:33:50.123 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 13, 2022-03-01T06:33:50.123 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 26, 2022-03-01T06:33:50.123 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 35, 2022-03-01T06:33:50.123 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 0, 2022-03-01T06:33:50.124 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 29, 2022-03-01T06:33:50.124 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:33:50.124 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:33:50.125 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.125 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.125 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.125 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6810", 2022-03-01T06:33:50.126 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.126 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.126 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.126 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.127 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6811", 2022-03-01T06:33:50.127 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.127 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.127 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.127 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.128 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:33:50.128 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.128 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.128 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.129 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6812", 2022-03-01T06:33:50.129 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.129 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.129 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.129 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.130 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6813", 2022-03-01T06:33:50.130 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.130 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.130 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.131 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.131 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:33:50.131 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.131 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.131 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.132 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6816", 2022-03-01T06:33:50.132 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.132 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.132 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.132 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.133 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6817", 2022-03-01T06:33:50.133 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.133 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.133 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.134 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.134 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:33:50.134 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:33:50.134 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.135 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:33:50.135 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6814", 2022-03-01T06:33:50.135 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.135 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.136 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.136 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:33:50.136 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6815", 2022-03-01T06:33:50.136 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 36842 2022-03-01T06:33:50.136 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.137 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.137 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.137 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6811/36842", 2022-03-01T06:33:50.137 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6813/36842", 2022-03-01T06:33:50.137 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6817/36842", 2022-03-01T06:33:50.138 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6815/36842", 2022-03-01T06:33:50.138 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:33:50.138 INFO:tasks.workunit.client.0.smithi169.stdout: "exists", 2022-03-01T06:33:50.139 INFO:tasks.workunit.client.0.smithi169.stdout: "up" 2022-03-01T06:33:50.139 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.139 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.139 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.139 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_xinfo": [ 2022-03-01T06:33:50.140 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.140 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:33:50.140 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:33:07.473308+0000", 2022-03-01T06:33:50.140 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:33:50.141 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:33:50.141 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:33:50.141 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:33:50.141 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:32:14.235597+0000", 2022-03-01T06:33:50.141 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:33:50.142 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.142 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.142 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:33:50.142 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:33:08.007518+0000", 2022-03-01T06:33:50.143 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:33:50.143 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:33:50.143 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:33:50.143 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:33:50.143 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:32:20.310039+0000", 2022-03-01T06:33:50.144 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:33:50.144 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.144 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.144 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:33:50.145 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:33:08.524791+0000", 2022-03-01T06:33:50.145 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:33:50.145 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:33:50.145 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:33:50.146 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:33:50.146 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:32:26.354142+0000", 2022-03-01T06:33:50.146 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:33:50.146 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.147 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.147 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_upmap": [], 2022-03-01T06:33:50.147 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_upmap_items": [], 2022-03-01T06:33:50.148 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_temp": [], 2022-03-01T06:33:50.148 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_temp": [], 2022-03-01T06:33:50.148 INFO:tasks.workunit.client.0.smithi169.stdout: "blacklist": {}, 2022-03-01T06:33:50.148 INFO:tasks.workunit.client.0.smithi169.stdout: "erasure_code_profiles": { 2022-03-01T06:33:50.149 INFO:tasks.workunit.client.0.smithi169.stdout: "default": { 2022-03-01T06:33:50.149 INFO:tasks.workunit.client.0.smithi169.stdout: "k": "2", 2022-03-01T06:33:50.149 INFO:tasks.workunit.client.0.smithi169.stdout: "m": "2", 2022-03-01T06:33:50.149 INFO:tasks.workunit.client.0.smithi169.stdout: "plugin": "jerasure", 2022-03-01T06:33:50.150 INFO:tasks.workunit.client.0.smithi169.stdout: "technique": "reed_sol_van" 2022-03-01T06:33:50.150 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.150 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.150 INFO:tasks.workunit.client.0.smithi169.stdout: "removed_snaps_queue": [], 2022-03-01T06:33:50.150 INFO:tasks.workunit.client.0.smithi169.stdout: "new_removed_snaps": [], 2022-03-01T06:33:50.151 INFO:tasks.workunit.client.0.smithi169.stdout: "new_purged_snaps": [], 2022-03-01T06:33:50.151 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_node_flags": {}, 2022-03-01T06:33:50.151 INFO:tasks.workunit.client.0.smithi169.stdout: "device_class_flags": {} 2022-03-01T06:33:50.151 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.152 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_metadata": [ 2022-03-01T06:33:50.152 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.152 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:33:50.152 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:33:50.153 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6804/36648,v1:127.0.0.1:6805/36648]", 2022-03-01T06:33:50.153 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:33:50.153 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:33:50.153 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:33:50.153 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:33:50.154 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:33:50.154 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:33:50.154 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:33:50.155 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:33:50.155 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:33:50.155 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2022-03-01T06:33:50.155 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:33:50.156 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:33:50.156 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:33:50.156 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:33:50.156 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:33:50.157 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:33:50.157 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:33:50.157 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:33:50.157 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:33:50.158 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:33:50.158 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:33:50.158 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:33:50.158 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:33:50.158 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:33:50.159 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:33:50.159 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6802/36648,v1:127.0.0.1:6803/36648]", 2022-03-01T06:33:50.159 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:33:50.159 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/36648,v1:127.0.0.1:6809/36648]", 2022-03-01T06:33:50.160 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/36648,v1:127.0.0.1:6807/36648]", 2022-03-01T06:33:50.160 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:33:50.160 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:33:50.160 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:33:50.160 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:33:50.161 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:33:50.161 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:33:50.161 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:33:50.161 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:33:50.162 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:33:50.162 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/0", 2022-03-01T06:33:50.162 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:33:50.162 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:33:50.163 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:33:50.163 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.163 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.163 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:33:50.163 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:33:50.164 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6812/32308,v1:127.0.0.1:6813/32308]", 2022-03-01T06:33:50.164 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:33:50.164 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:33:50.164 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:33:50.165 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:33:50.165 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:33:50.165 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:33:50.165 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:33:50.166 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:33:50.166 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:33:50.166 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2022-03-01T06:33:50.166 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:33:50.167 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:33:50.167 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:33:50.167 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:33:50.167 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:33:50.167 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:33:50.168 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:33:50.168 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:33:50.168 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:33:50.168 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:33:50.169 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:33:50.169 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:33:50.169 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:33:50.169 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:33:50.169 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:33:50.170 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6810/32308,v1:127.0.0.1:6811/32308]", 2022-03-01T06:33:50.170 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:33:50.170 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/32308,v1:127.0.0.1:6817/32308]", 2022-03-01T06:33:50.170 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/32308,v1:127.0.0.1:6815/32308]", 2022-03-01T06:33:50.171 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:33:50.171 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:33:50.171 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:33:50.171 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:33:50.171 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:33:50.172 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:33:50.172 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:33:50.172 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:33:50.172 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:33:50.173 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/1", 2022-03-01T06:33:50.173 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:33:50.173 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:33:50.173 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:33:50.173 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.174 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.174 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:33:50.174 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:33:50.174 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6812/36842,v1:127.0.0.1:6813/36842]", 2022-03-01T06:33:50.175 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:33:50.175 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:33:50.176 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:33:50.176 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:33:50.176 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:33:50.177 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:33:50.177 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:33:50.177 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:33:50.177 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:33:50.178 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2022-03-01T06:33:50.178 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:33:50.178 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:33:50.178 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:33:50.179 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:33:50.179 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:33:50.180 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:33:50.180 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:33:50.180 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:33:50.180 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:33:50.181 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:33:50.181 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:33:50.181 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:33:50.181 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:33:50.182 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:33:50.182 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:33:50.182 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6810/36842,v1:127.0.0.1:6811/36842]", 2022-03-01T06:33:50.182 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:33:50.183 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/36842,v1:127.0.0.1:6817/36842]", 2022-03-01T06:33:50.183 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/36842,v1:127.0.0.1:6815/36842]", 2022-03-01T06:33:50.183 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:33:50.183 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:33:50.184 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:33:50.184 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:33:50.184 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:33:50.184 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:33:50.185 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:33:50.185 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:33:50.185 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:33:50.185 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/2", 2022-03-01T06:33:50.185 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:33:50.186 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:33:50.186 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:33:50.186 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.187 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.187 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_clean_epochs": { 2022-03-01T06:33:50.187 INFO:tasks.workunit.client.0.smithi169.stdout: "min_last_epoch_clean": 0, 2022-03-01T06:33:50.187 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_clean": { 2022-03-01T06:33:50.187 INFO:tasks.workunit.client.0.smithi169.stdout: "per_pool": [] 2022-03-01T06:33:50.188 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.188 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_epochs": [ 2022-03-01T06:33:50.189 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.189 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:33:50.189 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 33 2022-03-01T06:33:50.189 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.190 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.190 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:33:50.190 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 9 2022-03-01T06:33:50.190 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.191 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.191 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:33:50.191 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 36 2022-03-01T06:33:50.191 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.192 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.192 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.192 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_first_committed": 1, 2022-03-01T06:33:50.192 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_last_committed": 38, 2022-03-01T06:33:50.192 INFO:tasks.workunit.client.0.smithi169.stdout: "crushmap": { 2022-03-01T06:33:50.193 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": [ 2022-03-01T06:33:50.193 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.193 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:33:50.193 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.0", 2022-03-01T06:33:50.194 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:33:50.194 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.194 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.194 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:33:50.195 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.1", 2022-03-01T06:33:50.195 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:33:50.195 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.196 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.196 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:33:50.196 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.2", 2022-03-01T06:33:50.196 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:33:50.196 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.197 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.197 INFO:tasks.workunit.client.0.smithi169.stdout: "types": [ 2022-03-01T06:33:50.197 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.198 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 0, 2022-03-01T06:33:50.198 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd" 2022-03-01T06:33:50.198 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.199 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.199 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:33:50.199 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "host" 2022-03-01T06:33:50.199 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.200 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.200 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 2, 2022-03-01T06:33:50.200 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "chassis" 2022-03-01T06:33:50.200 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.201 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.201 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 3, 2022-03-01T06:33:50.201 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "rack" 2022-03-01T06:33:50.201 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.202 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.202 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 4, 2022-03-01T06:33:50.202 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "row" 2022-03-01T06:33:50.203 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.203 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.204 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 5, 2022-03-01T06:33:50.204 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "pdu" 2022-03-01T06:33:50.204 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.204 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.205 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 6, 2022-03-01T06:33:50.205 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "pod" 2022-03-01T06:33:50.205 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.205 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.206 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 7, 2022-03-01T06:33:50.206 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "room" 2022-03-01T06:33:50.206 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.206 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.207 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 8, 2022-03-01T06:33:50.207 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "datacenter" 2022-03-01T06:33:50.207 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.207 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.207 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 9, 2022-03-01T06:33:50.208 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "zone" 2022-03-01T06:33:50.208 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.208 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.208 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 10, 2022-03-01T06:33:50.209 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "region" 2022-03-01T06:33:50.209 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.209 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.209 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:33:50.210 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "root" 2022-03-01T06:33:50.210 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.210 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.210 INFO:tasks.workunit.client.0.smithi169.stdout: "buckets": [ 2022-03-01T06:33:50.211 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.211 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -1, 2022-03-01T06:33:50.211 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "default", 2022-03-01T06:33:50.211 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:33:50.211 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "root", 2022-03-01T06:33:50.212 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:33:50.212 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:33:50.212 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:33:50.212 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:33:50.213 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.213 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -3, 2022-03-01T06:33:50.213 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:33:50.213 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:33:50.213 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.214 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.214 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.214 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.214 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -2, 2022-03-01T06:33:50.215 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "default~hdd", 2022-03-01T06:33:50.215 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:33:50.215 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "root", 2022-03-01T06:33:50.215 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:33:50.215 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:33:50.216 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:33:50.216 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:33:50.216 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.216 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -4, 2022-03-01T06:33:50.216 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:33:50.217 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:33:50.217 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.217 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.217 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.218 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.218 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -3, 2022-03-01T06:33:50.218 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "smithi169", 2022-03-01T06:33:50.218 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:33:50.218 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "host", 2022-03-01T06:33:50.219 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:33:50.219 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:33:50.219 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:33:50.219 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:33:50.220 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.220 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:33:50.220 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:33:50.220 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:33:50.220 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.221 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.221 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:33:50.221 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:33:50.221 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 1 2022-03-01T06:33:50.222 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.222 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.222 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:33:50.222 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:33:50.222 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 2 2022-03-01T06:33:50.223 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.223 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.223 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.223 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.224 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -4, 2022-03-01T06:33:50.224 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "smithi169~hdd", 2022-03-01T06:33:50.224 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:33:50.224 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "host", 2022-03-01T06:33:50.224 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:33:50.225 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:33:50.225 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:33:50.225 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:33:50.225 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.226 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:33:50.226 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:33:50.226 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:33:50.226 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.227 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.227 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:33:50.227 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:33:50.227 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 1 2022-03-01T06:33:50.227 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.228 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.228 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:33:50.228 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:33:50.228 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 2 2022-03-01T06:33:50.228 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.229 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.229 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.229 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.229 INFO:tasks.workunit.client.0.smithi169.stdout: "rules": [ 2022-03-01T06:33:50.230 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.230 INFO:tasks.workunit.client.0.smithi169.stdout: "rule_id": 0, 2022-03-01T06:33:50.230 INFO:tasks.workunit.client.0.smithi169.stdout: "rule_name": "replicated_rule", 2022-03-01T06:33:50.230 INFO:tasks.workunit.client.0.smithi169.stdout: "ruleset": 0, 2022-03-01T06:33:50.231 INFO:tasks.workunit.client.0.smithi169.stdout: "type": 1, 2022-03-01T06:33:50.231 INFO:tasks.workunit.client.0.smithi169.stdout: "min_size": 1, 2022-03-01T06:33:50.231 INFO:tasks.workunit.client.0.smithi169.stdout: "max_size": 10, 2022-03-01T06:33:50.231 INFO:tasks.workunit.client.0.smithi169.stdout: "steps": [ 2022-03-01T06:33:50.232 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.232 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "take", 2022-03-01T06:33:50.232 INFO:tasks.workunit.client.0.smithi169.stdout: "item": -1, 2022-03-01T06:33:50.232 INFO:tasks.workunit.client.0.smithi169.stdout: "item_name": "default" 2022-03-01T06:33:50.232 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.233 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.233 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "choose_firstn", 2022-03-01T06:33:50.233 INFO:tasks.workunit.client.0.smithi169.stdout: "num": 0, 2022-03-01T06:33:50.233 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "osd" 2022-03-01T06:33:50.234 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.234 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.234 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "emit" 2022-03-01T06:33:50.234 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.234 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:33:50.235 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.235 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.235 INFO:tasks.workunit.client.0.smithi169.stdout: "tunables": { 2022-03-01T06:33:50.235 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_local_tries": 0, 2022-03-01T06:33:50.236 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_local_fallback_tries": 0, 2022-03-01T06:33:50.236 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_total_tries": 50, 2022-03-01T06:33:50.236 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_descend_once": 1, 2022-03-01T06:33:50.236 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_vary_r": 1, 2022-03-01T06:33:50.236 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_stable": 1, 2022-03-01T06:33:50.237 INFO:tasks.workunit.client.0.smithi169.stdout: "straw_calc_version": 1, 2022-03-01T06:33:50.237 INFO:tasks.workunit.client.0.smithi169.stdout: "allowed_bucket_algs": 54, 2022-03-01T06:33:50.237 INFO:tasks.workunit.client.0.smithi169.stdout: "profile": "jewel", 2022-03-01T06:33:50.237 INFO:tasks.workunit.client.0.smithi169.stdout: "optimal_tunables": 1, 2022-03-01T06:33:50.238 INFO:tasks.workunit.client.0.smithi169.stdout: "legacy_tunables": 0, 2022-03-01T06:33:50.238 INFO:tasks.workunit.client.0.smithi169.stdout: "minimum_required_version": "jewel", 2022-03-01T06:33:50.238 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables": 1, 2022-03-01T06:33:50.238 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables2": 1, 2022-03-01T06:33:50.239 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v2_rules": 0, 2022-03-01T06:33:50.239 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables3": 1, 2022-03-01T06:33:50.239 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v3_rules": 0, 2022-03-01T06:33:50.239 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v4_buckets": 1, 2022-03-01T06:33:50.239 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables5": 1, 2022-03-01T06:33:50.240 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v5_rules": 0 2022-03-01T06:33:50.240 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.240 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_args": {} 2022-03-01T06:33:50.240 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.241 INFO:tasks.workunit.client.0.smithi169.stdout: "fsmap": { 2022-03-01T06:33:50.241 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 1, 2022-03-01T06:33:50.241 INFO:tasks.workunit.client.0.smithi169.stdout: "default_fscid": -1, 2022-03-01T06:33:50.241 INFO:tasks.workunit.client.0.smithi169.stdout: "compat": { 2022-03-01T06:33:50.241 INFO:tasks.workunit.client.0.smithi169.stdout: "compat": {}, 2022-03-01T06:33:50.242 INFO:tasks.workunit.client.0.smithi169.stdout: "ro_compat": {}, 2022-03-01T06:33:50.242 INFO:tasks.workunit.client.0.smithi169.stdout: "incompat": { 2022-03-01T06:33:50.242 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_1": "base v0.20", 2022-03-01T06:33:50.242 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_2": "client writeable ranges", 2022-03-01T06:33:50.243 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_3": "default file layouts on dirs", 2022-03-01T06:33:50.243 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_4": "dir inode in separate object", 2022-03-01T06:33:50.243 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_5": "mds uses versioned encoding", 2022-03-01T06:33:50.243 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_6": "dirfrag is stored in omap", 2022-03-01T06:33:50.243 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_8": "no anchor table", 2022-03-01T06:33:50.244 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_9": "file layout v2", 2022-03-01T06:33:50.244 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_10": "snaprealm v2" 2022-03-01T06:33:50.244 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.244 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.245 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_flags": { 2022-03-01T06:33:50.245 INFO:tasks.workunit.client.0.smithi169.stdout: "enable_multiple": false, 2022-03-01T06:33:50.245 INFO:tasks.workunit.client.0.smithi169.stdout: "ever_enabled_multiple": false 2022-03-01T06:33:50.245 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.246 INFO:tasks.workunit.client.0.smithi169.stdout: "standbys": [], 2022-03-01T06:33:50.246 INFO:tasks.workunit.client.0.smithi169.stdout: "filesystems": [] 2022-03-01T06:33:50.246 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.246 INFO:tasks.workunit.client.0.smithi169.stdout: "mdsmap_first_committed": 1, 2022-03-01T06:33:50.246 INFO:tasks.workunit.client.0.smithi169.stdout: "mdsmap_last_committed": 1, 2022-03-01T06:33:50.247 INFO:tasks.workunit.client.0.smithi169.stdout: "auth": { 2022-03-01T06:33:50.247 INFO:tasks.workunit.client.0.smithi169.stdout: "first_committed": 1, 2022-03-01T06:33:50.247 INFO:tasks.workunit.client.0.smithi169.stdout: "last_committed": 5, 2022-03-01T06:33:50.247 INFO:tasks.workunit.client.0.smithi169.stdout: "num_secrets": 3 2022-03-01T06:33:50.248 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.248 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg": 1, 2022-03-01T06:33:50.248 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_active": 1, 2022-03-01T06:33:50.248 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_unknown": 0, 2022-03-01T06:33:50.249 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osd": 3, 2022-03-01T06:33:50.249 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_sum": { 2022-03-01T06:33:50.249 INFO:tasks.workunit.client.0.smithi169.stdout: "stat_sum": { 2022-03-01T06:33:50.249 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes": 48300, 2022-03-01T06:33:50.250 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects": 100, 2022-03-01T06:33:50.250 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_clones": 0, 2022-03-01T06:33:50.250 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_copies": 300, 2022-03-01T06:33:50.250 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing_on_primary": 0, 2022-03-01T06:33:50.251 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing": 0, 2022-03-01T06:33:50.251 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_degraded": 100, 2022-03-01T06:33:50.251 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_misplaced": 0, 2022-03-01T06:33:50.251 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_unfound": 0, 2022-03-01T06:33:50.252 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_dirty": 100, 2022-03-01T06:33:50.252 INFO:tasks.workunit.client.0.smithi169.stdout: "num_whiteouts": 0, 2022-03-01T06:33:50.252 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read": 0, 2022-03-01T06:33:50.252 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read_kb": 0, 2022-03-01T06:33:50.252 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write": 100, 2022-03-01T06:33:50.253 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write_kb": 100, 2022-03-01T06:33:50.253 INFO:tasks.workunit.client.0.smithi169.stdout: "num_scrub_errors": 0, 2022-03-01T06:33:50.253 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shallow_scrub_errors": 0, 2022-03-01T06:33:50.253 INFO:tasks.workunit.client.0.smithi169.stdout: "num_deep_scrub_errors": 0, 2022-03-01T06:33:50.254 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_recovered": 0, 2022-03-01T06:33:50.254 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_recovered": 0, 2022-03-01T06:33:50.254 INFO:tasks.workunit.client.0.smithi169.stdout: "num_keys_recovered": 0, 2022-03-01T06:33:50.254 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_omap": 0, 2022-03-01T06:33:50.255 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_hit_set_archive": 0, 2022-03-01T06:33:50.255 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_hit_set_archive": 0, 2022-03-01T06:33:50.255 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush": 0, 2022-03-01T06:33:50.255 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_kb": 0, 2022-03-01T06:33:50.255 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict": 0, 2022-03-01T06:33:50.256 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_kb": 0, 2022-03-01T06:33:50.256 INFO:tasks.workunit.client.0.smithi169.stdout: "num_promote": 0, 2022-03-01T06:33:50.256 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_high": 0, 2022-03-01T06:33:50.256 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_low": 0, 2022-03-01T06:33:50.257 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_some": 0, 2022-03-01T06:33:50.257 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_full": 0, 2022-03-01T06:33:50.257 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_pinned": 0, 2022-03-01T06:33:50.257 INFO:tasks.workunit.client.0.smithi169.stdout: "num_legacy_snapsets": 0, 2022-03-01T06:33:50.257 INFO:tasks.workunit.client.0.smithi169.stdout: "num_large_omap_objects": 0, 2022-03-01T06:33:50.258 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_manifest": 0, 2022-03-01T06:33:50.258 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_bytes": 0, 2022-03-01T06:33:50.258 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_keys": 0, 2022-03-01T06:33:50.258 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_repaired": 0 2022-03-01T06:33:50.259 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.259 INFO:tasks.workunit.client.0.smithi169.stdout: "store_stats": { 2022-03-01T06:33:50.259 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 0, 2022-03-01T06:33:50.259 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 0, 2022-03-01T06:33:50.259 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 0, 2022-03-01T06:33:50.260 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 0, 2022-03-01T06:33:50.260 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 0, 2022-03-01T06:33:50.260 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:33:50.260 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:33:50.261 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:33:50.261 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 0, 2022-03-01T06:33:50.261 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 0 2022-03-01T06:33:50.261 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.261 INFO:tasks.workunit.client.0.smithi169.stdout: "log_size": 16, 2022-03-01T06:33:50.262 INFO:tasks.workunit.client.0.smithi169.stdout: "ondisk_log_size": 16, 2022-03-01T06:33:50.262 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 2, 2022-03-01T06:33:50.262 INFO:tasks.workunit.client.0.smithi169.stdout: "acting": 2, 2022-03-01T06:33:50.262 INFO:tasks.workunit.client.0.smithi169.stdout: "num_store_stats": 0 2022-03-01T06:33:50.263 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.263 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_sum": { 2022-03-01T06:33:50.263 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 0, 2022-03-01T06:33:50.263 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 0, 2022-03-01T06:33:50.263 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pgs": 3, 2022-03-01T06:33:50.264 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osds": 3, 2022-03-01T06:33:50.264 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_osds": 3, 2022-03-01T06:33:50.264 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_omap_osds": 3, 2022-03-01T06:33:50.264 INFO:tasks.workunit.client.0.smithi169.stdout: "kb": 314572800, 2022-03-01T06:33:50.265 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used": 3178624, 2022-03-01T06:33:50.265 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_data": 32640, 2022-03-01T06:33:50.265 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_omap": 11, 2022-03-01T06:33:50.265 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_meta": 3145716, 2022-03-01T06:33:50.265 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_avail": 311394176, 2022-03-01T06:33:50.266 INFO:tasks.workunit.client.0.smithi169.stdout: "statfs": { 2022-03-01T06:33:50.266 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 322122547200, 2022-03-01T06:33:50.266 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 318867636224, 2022-03-01T06:33:50.266 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 3221225472, 2022-03-01T06:33:50.267 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 33423360, 2022-03-01T06:33:50.267 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 400942, 2022-03-01T06:33:50.267 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:33:50.267 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:33:50.268 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:33:50.268 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 12052, 2022-03-01T06:33:50.268 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 3221213420 2022-03-01T06:33:50.268 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.268 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_peers": [], 2022-03-01T06:33:50.269 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_trim_queue_len": 0, 2022-03-01T06:33:50.269 INFO:tasks.workunit.client.0.smithi169.stdout: "num_snap_trimming": 0, 2022-03-01T06:33:50.269 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shards_repaired": 1, 2022-03-01T06:33:50.269 INFO:tasks.workunit.client.0.smithi169.stdout: "op_queue_age_hist": { 2022-03-01T06:33:50.270 INFO:tasks.workunit.client.0.smithi169.stdout: "histogram": [], 2022-03-01T06:33:50.270 INFO:tasks.workunit.client.0.smithi169.stdout: "upper_bound": 1 2022-03-01T06:33:50.270 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.270 INFO:tasks.workunit.client.0.smithi169.stdout: "perf_stat": { 2022-03-01T06:33:50.270 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ms": 0, 2022-03-01T06:33:50.271 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ms": 0, 2022-03-01T06:33:50.271 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ns": 0, 2022-03-01T06:33:50.271 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ns": 0 2022-03-01T06:33:50.271 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.272 INFO:tasks.workunit.client.0.smithi169.stdout: "alerts": [], 2022-03-01T06:33:50.272 INFO:tasks.workunit.client.0.smithi169.stdout: "network_ping_times": [] 2022-03-01T06:33:50.272 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.272 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_sum_by_class": { 2022-03-01T06:33:50.273 INFO:tasks.workunit.client.0.smithi169.stdout: "hdd": { 2022-03-01T06:33:50.273 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 0, 2022-03-01T06:33:50.273 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 0, 2022-03-01T06:33:50.273 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pgs": 3, 2022-03-01T06:33:50.274 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osds": 3, 2022-03-01T06:33:50.274 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_osds": 3, 2022-03-01T06:33:50.274 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_omap_osds": 3, 2022-03-01T06:33:50.274 INFO:tasks.workunit.client.0.smithi169.stdout: "kb": 314572800, 2022-03-01T06:33:50.274 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used": 3178624, 2022-03-01T06:33:50.275 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_data": 32640, 2022-03-01T06:33:50.275 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_omap": 11, 2022-03-01T06:33:50.275 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_meta": 3145716, 2022-03-01T06:33:50.275 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_avail": 311394176, 2022-03-01T06:33:50.276 INFO:tasks.workunit.client.0.smithi169.stdout: "statfs": { 2022-03-01T06:33:50.276 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 322122547200, 2022-03-01T06:33:50.276 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 318867636224, 2022-03-01T06:33:50.276 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 3221225472, 2022-03-01T06:33:50.277 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 33423360, 2022-03-01T06:33:50.277 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 400942, 2022-03-01T06:33:50.277 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:33:50.277 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:33:50.277 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:33:50.278 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 12052, 2022-03-01T06:33:50.278 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 3221213420 2022-03-01T06:33:50.278 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.278 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_peers": [], 2022-03-01T06:33:50.278 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_trim_queue_len": 0, 2022-03-01T06:33:50.279 INFO:tasks.workunit.client.0.smithi169.stdout: "num_snap_trimming": 0, 2022-03-01T06:33:50.279 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shards_repaired": 1, 2022-03-01T06:33:50.279 INFO:tasks.workunit.client.0.smithi169.stdout: "op_queue_age_hist": { 2022-03-01T06:33:50.279 INFO:tasks.workunit.client.0.smithi169.stdout: "histogram": [], 2022-03-01T06:33:50.280 INFO:tasks.workunit.client.0.smithi169.stdout: "upper_bound": 1 2022-03-01T06:33:50.280 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.280 INFO:tasks.workunit.client.0.smithi169.stdout: "perf_stat": { 2022-03-01T06:33:50.280 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ms": 0, 2022-03-01T06:33:50.280 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ms": 0, 2022-03-01T06:33:50.281 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ns": 0, 2022-03-01T06:33:50.281 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ns": 0 2022-03-01T06:33:50.281 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.281 INFO:tasks.workunit.client.0.smithi169.stdout: "alerts": [], 2022-03-01T06:33:50.282 INFO:tasks.workunit.client.0.smithi169.stdout: "network_ping_times": [] 2022-03-01T06:33:50.282 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.282 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.282 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_stats": [ 2022-03-01T06:33:50.282 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.283 INFO:tasks.workunit.client.0.smithi169.stdout: "poolid": 1, 2022-03-01T06:33:50.283 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg": 1, 2022-03-01T06:33:50.283 INFO:tasks.workunit.client.0.smithi169.stdout: "stat_sum": { 2022-03-01T06:33:50.283 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes": 48300, 2022-03-01T06:33:50.284 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects": 100, 2022-03-01T06:33:50.284 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_clones": 0, 2022-03-01T06:33:50.284 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_copies": 300, 2022-03-01T06:33:50.284 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing_on_primary": 0, 2022-03-01T06:33:50.284 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing": 0, 2022-03-01T06:33:50.285 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_degraded": 100, 2022-03-01T06:33:50.285 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_misplaced": 0, 2022-03-01T06:33:50.285 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_unfound": 0, 2022-03-01T06:33:50.285 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_dirty": 100, 2022-03-01T06:33:50.286 INFO:tasks.workunit.client.0.smithi169.stdout: "num_whiteouts": 0, 2022-03-01T06:33:50.286 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read": 0, 2022-03-01T06:33:50.286 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read_kb": 0, 2022-03-01T06:33:50.286 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write": 100, 2022-03-01T06:33:50.287 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write_kb": 100, 2022-03-01T06:33:50.287 INFO:tasks.workunit.client.0.smithi169.stdout: "num_scrub_errors": 0, 2022-03-01T06:33:50.287 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shallow_scrub_errors": 0, 2022-03-01T06:33:50.287 INFO:tasks.workunit.client.0.smithi169.stdout: "num_deep_scrub_errors": 0, 2022-03-01T06:33:50.287 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_recovered": 0, 2022-03-01T06:33:50.288 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_recovered": 0, 2022-03-01T06:33:50.288 INFO:tasks.workunit.client.0.smithi169.stdout: "num_keys_recovered": 0, 2022-03-01T06:33:50.288 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_omap": 0, 2022-03-01T06:33:50.288 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_hit_set_archive": 0, 2022-03-01T06:33:50.289 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_hit_set_archive": 0, 2022-03-01T06:33:50.289 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush": 0, 2022-03-01T06:33:50.289 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_kb": 0, 2022-03-01T06:33:50.289 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict": 0, 2022-03-01T06:33:50.290 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_kb": 0, 2022-03-01T06:33:50.290 INFO:tasks.workunit.client.0.smithi169.stdout: "num_promote": 0, 2022-03-01T06:33:50.290 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_high": 0, 2022-03-01T06:33:50.290 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_low": 0, 2022-03-01T06:33:50.290 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_some": 0, 2022-03-01T06:33:50.291 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_full": 0, 2022-03-01T06:33:50.291 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_pinned": 0, 2022-03-01T06:33:50.291 INFO:tasks.workunit.client.0.smithi169.stdout: "num_legacy_snapsets": 0, 2022-03-01T06:33:50.291 INFO:tasks.workunit.client.0.smithi169.stdout: "num_large_omap_objects": 0, 2022-03-01T06:33:50.292 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_manifest": 0, 2022-03-01T06:33:50.292 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_bytes": 0, 2022-03-01T06:33:50.292 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_keys": 0, 2022-03-01T06:33:50.292 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_repaired": 0 2022-03-01T06:33:50.292 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.293 INFO:tasks.workunit.client.0.smithi169.stdout: "store_stats": { 2022-03-01T06:33:50.293 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 0, 2022-03-01T06:33:50.293 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 0, 2022-03-01T06:33:50.293 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 0, 2022-03-01T06:33:50.294 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 19660800, 2022-03-01T06:33:50.294 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 144900, 2022-03-01T06:33:50.294 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:33:50.294 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:33:50.295 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:33:50.295 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 0, 2022-03-01T06:33:50.295 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 0 2022-03-01T06:33:50.295 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.296 INFO:tasks.workunit.client.0.smithi169.stdout: "log_size": 16, 2022-03-01T06:33:50.296 INFO:tasks.workunit.client.0.smithi169.stdout: "ondisk_log_size": 16, 2022-03-01T06:33:50.296 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 2, 2022-03-01T06:33:50.296 INFO:tasks.workunit.client.0.smithi169.stdout: "acting": 2, 2022-03-01T06:33:50.297 INFO:tasks.workunit.client.0.smithi169.stdout: "num_store_stats": 3 2022-03-01T06:33:50.297 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.297 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.297 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_stats": [ 2022-03-01T06:33:50.298 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.298 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:33:50.298 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 141733920778 2022-03-01T06:33:50.298 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.299 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.299 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:33:50.299 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 38654705680 2022-03-01T06:33:50.299 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.299 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.300 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:33:50.300 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 150323855370 2022-03-01T06:33:50.300 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.300 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.300 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_by_state": [ 2022-03-01T06:33:50.301 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.301 INFO:tasks.workunit.client.0.smithi169.stdout: "state": "active+undersized+degraded", 2022-03-01T06:33:50.301 INFO:tasks.workunit.client.0.smithi169.stdout: "num": 1 2022-03-01T06:33:50.301 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.302 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.302 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_by_osd": [ 2022-03-01T06:33:50.302 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.302 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:33:50.302 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 0, 2022-03-01T06:33:50.303 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 1, 2022-03-01T06:33:50.303 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:33:50.303 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.303 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.304 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:33:50.304 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 0, 2022-03-01T06:33:50.304 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 0, 2022-03-01T06:33:50.304 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:33:50.304 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.305 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.305 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:33:50.305 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 1, 2022-03-01T06:33:50.305 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 1, 2022-03-01T06:33:50.306 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:33:50.306 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.306 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.306 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snaps": [ 2022-03-01T06:33:50.306 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:33:50.307 INFO:tasks.workunit.client.0.smithi169.stdout: "pool": 1, 2022-03-01T06:33:50.307 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snaps": {} 2022-03-01T06:33:50.307 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.307 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:33:50.308 INFO:tasks.workunit.client.0.smithi169.stdout: "servicemap": { 2022-03-01T06:33:50.308 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 3, 2022-03-01T06:33:50.308 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:33:21.385057+0000", 2022-03-01T06:33:50.308 INFO:tasks.workunit.client.0.smithi169.stdout: "services": { 2022-03-01T06:33:50.308 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": { 2022-03-01T06:33:50.309 INFO:tasks.workunit.client.0.smithi169.stdout: "daemons": { 2022-03-01T06:33:50.309 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": "", 2022-03-01T06:33:50.309 INFO:tasks.workunit.client.0.smithi169.stdout: "0": { 2022-03-01T06:33:50.309 INFO:tasks.workunit.client.0.smithi169.stdout: "start_epoch": 0, 2022-03-01T06:33:50.310 INFO:tasks.workunit.client.0.smithi169.stdout: "start_stamp": "0.000000", 2022-03-01T06:33:50.310 INFO:tasks.workunit.client.0.smithi169.stdout: "gid": 0, 2022-03-01T06:33:50.310 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "(unrecognized address family 0)/0", 2022-03-01T06:33:50.310 INFO:tasks.workunit.client.0.smithi169.stdout: "metadata": {}, 2022-03-01T06:33:50.310 INFO:tasks.workunit.client.0.smithi169.stdout: "task_status": {} 2022-03-01T06:33:50.311 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.311 INFO:tasks.workunit.client.0.smithi169.stdout: "2": { 2022-03-01T06:33:50.311 INFO:tasks.workunit.client.0.smithi169.stdout: "start_epoch": 0, 2022-03-01T06:33:50.311 INFO:tasks.workunit.client.0.smithi169.stdout: "start_stamp": "0.000000", 2022-03-01T06:33:50.312 INFO:tasks.workunit.client.0.smithi169.stdout: "gid": 0, 2022-03-01T06:33:50.312 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "(unrecognized address family 0)/0", 2022-03-01T06:33:50.312 INFO:tasks.workunit.client.0.smithi169.stdout: "metadata": {}, 2022-03-01T06:33:50.312 INFO:tasks.workunit.client.0.smithi169.stdout: "task_status": {} 2022-03-01T06:33:50.312 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.313 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.313 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.313 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.313 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:33:50.314 INFO:tasks.workunit.client.0.smithi169.stdout: "paxos": { 2022-03-01T06:33:50.314 INFO:tasks.workunit.client.0.smithi169.stdout: "first_committed": 1, 2022-03-01T06:33:50.314 INFO:tasks.workunit.client.0.smithi169.stdout: "last_committed": 337, 2022-03-01T06:33:50.314 INFO:tasks.workunit.client.0.smithi169.stdout: "last_pn": 0, 2022-03-01T06:33:50.315 INFO:tasks.workunit.client.0.smithi169.stdout: "accepted_pn": 0 2022-03-01T06:33:50.315 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:33:50.315 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:33:50.316 INFO:tasks.workunit.client.0.smithi169.stderr:report 2567956003 2022-03-01T06:33:50.316 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2022-03-01T06:33:50.317 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2022-03-01T06:33:50.317 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2022-03-01T06:33:50.317 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2022-03-01T06:33:50.318 INFO:tasks.workunit.client.0.smithi169.stdout:writing non-divergent object existing_4 2022-03-01T06:33:50.318 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:50.328 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:50.329 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2022-03-01T06:33:15.473049+0000 26'100 38:121 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:50.329 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:50.329 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:50.331 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:50.339 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2022-03-01T06:33:50.428 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2022-03-01T06:33:50.429 INFO:tasks.workunit.client.0.smithi169.stdout:delay recovery 2022-03-01T06:33:50.429 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:50.729 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:50.729 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2022-03-01T06:33:15.473049+0000 26'100 38:121 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:50.729 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:50.730 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:50.730 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:50.739 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:50.740 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2022-03-01T06:33:50.740 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-03-01T06:33:50.740 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-03-01T06:33:50.741 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:33:50.741 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:33:50.741 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:33:50.741 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.0.asok 2022-03-01T06:33:50.742 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2022-03-01T06:33:50.742 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.0.asok set_recovery_delay 100000 2022-03-01T06:33:50.826 INFO:tasks.workunit.client.0.smithi169.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-03-01T06:33:50.834 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2022-03-01T06:33:50.834 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2022-03-01T06:33:50.834 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-03-01T06:33:50.835 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-03-01T06:33:50.835 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:33:50.835 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:33:50.835 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:33:50.836 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.2.asok 2022-03-01T06:33:50.836 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2022-03-01T06:33:50.836 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.2.asok set_recovery_delay 100000 2022-03-01T06:33:50.920 INFO:tasks.workunit.client.0.smithi169.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-03-01T06:33:50.928 INFO:tasks.workunit.client.0.smithi169.stdout:revive divergent 1 2022-03-01T06:33:50.928 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2022-03-01T06:33:50.928 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:51.229 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:51.230 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2022-03-01T06:33:15.473049+0000 26'100 38:121 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:51.230 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:51.230 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:51.231 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:51.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2022-03-01T06:33:51.865 INFO:tasks.workunit.client.0.smithi169.stderr:noup is set 2022-03-01T06:33:51.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2022-03-01T06:33:51.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:33:51.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:33:51.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-03-01T06:33:51.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:33:51.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:33:51.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:33:51.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:33:51.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:33:51.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:33:51.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:33:51.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:33:51.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:33:51.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:33:51.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:33:51.880 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:33:51.880 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:33:51.880 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:33:51.881 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:33:51.881 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:33:51.881 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:33:51.881 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:33:51.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:33:51.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:33:51.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:33:51.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:33:51.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:33:51.883 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:33:51.883 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:33:51.883 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:33:51.883 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:33:51.883 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:33:51.884 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:33:51.885 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-03-01T06:33:51.885 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:33:51.886 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-03-01T06:33:51.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-03-01T06:33:51.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:33:51.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:33:51.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:33:51.903 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:51.922+0000 7ff59bfcdf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:51.905 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:51.924+0000 7ff59bfcdf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:51.906 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:51.925+0000 7ff59bfcdf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:33:52.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2022-03-01T06:33:52.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2022-03-01T06:33:52.422 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:52.441+0000 7ff59bfcdf00 -1 Falling back to public interface 2022-03-01T06:33:53.245 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:33:53.265+0000 7ff59bfcdf00 -1 osd.1 26 log_to_monitors {default=true} 2022-03-01T06:33:57.203 INFO:tasks.workunit.client.0.smithi169.stdout:delay recovery divergent 2022-03-01T06:33:57.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2022-03-01T06:33:57.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:57.517 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:57.518 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2022-03-01T06:33:15.473049+0000 38'101 40:125 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:57.518 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:57.518 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:57.519 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:57.528 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2022-03-01T06:33:57.529 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-03-01T06:33:57.529 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-03-01T06:33:57.529 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:33:57.529 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:33:57.530 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:33:57.530 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.1.asok 2022-03-01T06:33:57.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2022-03-01T06:33:57.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.1.asok set_recovery_delay 100000 2022-03-01T06:33:57.615 INFO:tasks.workunit.client.0.smithi169.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-03-01T06:33:57.622 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2022-03-01T06:33:58.196 INFO:tasks.workunit.client.0.smithi169.stderr:noup is unset 2022-03-01T06:33:58.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2022-03-01T06:33:58.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:33:58.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:33:58.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:33:58.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:33:58.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:58.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:33:58.209 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:33:58.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:58.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:33:58.605 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 33 up_thru 37 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/36648,v1:127.0.0.1:6803/36648] [v2:127.0.0.1:6804/36648,v1:127.0.0.1:6805/36648] exists,up b36cc591-5c05-4344-a0c6-fd7c20be6391 2022-03-01T06:33:58.607 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:33:58.607 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:33:58.608 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:33:58.608 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2022-03-01T06:33:58.608 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:33:58.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:33:58.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:33:58.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:33:58.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:58.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:33:58.610 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:33:58.610 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:58.611 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:33:58.933 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 28 last_clean_interval [9,26) [v2:127.0.0.1:6818/38304,v1:127.0.0.1:6819/38304] [v2:127.0.0.1:6820/38304,v1:127.0.0.1:6821/38304] exists,up fd55fbe2-fa36-4813-92e3-6da6f78c70a0 2022-03-01T06:33:58.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:33:58.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:33:58.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:33:58.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2022-03-01T06:33:58.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:33:58.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:33:58.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:33:58.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:33:58.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:33:58.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:33:58.938 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:33:58.938 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:33:58.939 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:33:59.257 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [13,26) [v2:127.0.0.1:6810/36842,v1:127.0.0.1:6811/36842] [v2:127.0.0.1:6812/36842,v1:127.0.0.1:6813/36842] exists,up 9d15391e-aa51-4a5b-868e-961f98e20902 2022-03-01T06:33:59.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:33:59.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:33:59.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:33:59.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:59.568 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:59.568 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2022-03-01T06:33:58.307765+0000 26'101 43:125 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:59.569 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:59.569 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:59.571 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:59.578 INFO:tasks.workunit.client.0.smithi169.stdout:wait for peering 2022-03-01T06:33:59.578 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2022-03-01T06:33:59.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2022-03-01T06:33:59.879 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:33:59.879 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2022-03-01T06:33:58.307765+0000 26'101 43:125 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:33:59.879 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:33:59.880 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:33:59.880 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:33:59.888 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2022-03-01T06:34:00.016 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2022-03-01T06:34:00.017 INFO:tasks.workunit.client.0.smithi169.stdout:killing divergent 1 2022-03-01T06:34:00.018 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2022-03-01T06:34:00.324 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:34:00.324 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2022-03-01T06:33:58.307765+0000 26'101 43:125 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:34:00.325 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:34:00.325 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:34:00.326 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:34:00.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2022-03-01T06:34:00.335 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:34:00.335 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:34:00.335 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:34:00.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:34:00.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:34:00.440 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:34:00.440 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2022-03-01T06:34:00.441 INFO:tasks.workunit.client.0.smithi169.stdout:reviving divergent 1 2022-03-01T06:34:00.441 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2022-03-01T06:34:00.744 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:34:00.744 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2022-03-01T06:33:58.307765+0000 26'101 43:125 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:34:00.745 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:34:00.745 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:34:00.746 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:34:00.758 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2022-03-01T06:34:00.758 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:34:00.758 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:34:00.758 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-03-01T06:34:00.759 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:34:00.759 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:34:00.759 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:34:00.759 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:34:00.760 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:34:00.760 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:34:00.760 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:34:00.760 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:34:00.761 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:34:00.761 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:34:00.761 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:34:00.761 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:34:00.761 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:34:00.762 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:34:00.762 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:34:00.762 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:00.763 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:00.763 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:00.763 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:00.763 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:34:00.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:34:00.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:34:00.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:34:00.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:34:00.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:00.765 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:34:00.765 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:34:00.765 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:34:00.765 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-03-01T06:34:00.766 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:34:00.767 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:00.767 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-03-01T06:34:00.767 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-03-01T06:34:00.768 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:34:00.768 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:34:00.768 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:34:00.784 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:00.804+0000 7f6c9f9c1f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:00.786 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:00.806+0000 7f6c9f9c1f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:00.787 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:00.807+0000 7f6c9f9c1f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:01.085 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2022-03-01T06:34:01.085 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:34:01.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:34:01.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:34:01.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:34:01.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:01.087 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:34:01.087 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:34:01.088 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:01.088 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:34:01.303 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:01.323+0000 7f6c9f9c1f00 -1 Falling back to public interface 2022-03-01T06:34:01.411 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 28 last_clean_interval [9,26) [v2:127.0.0.1:6818/38304,v1:127.0.0.1:6819/38304] [v2:127.0.0.1:6820/38304,v1:127.0.0.1:6821/38304] exists,up fd55fbe2-fa36-4813-92e3-6da6f78c70a0 2022-03-01T06:34:01.412 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:34:01.412 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:34:01.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:34:01.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2022-03-01T06:34:02.193 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:02.213+0000 7f6c9f9c1f00 -1 osd.1 44 log_to_monitors {default=true} 2022-03-01T06:34:21.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2022-03-01T06:34:21.414 INFO:tasks.workunit.client.0.smithi169.stdout:allowing recovery 2022-03-01T06:34:21.414 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2022-03-01T06:34:21.727 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:34:21.727 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2022-03-01T06:34:03.546614+0000 44'102 48:150 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:34:21.728 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:34:21.728 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:34:21.729 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:34:21.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2022-03-01T06:34:21.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2022-03-01T06:34:21.853 INFO:tasks.workunit.client.0.smithi169.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-03-01T06:34:21.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2022-03-01T06:34:21.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2022-03-01T06:34:21.975 INFO:tasks.workunit.client.0.smithi169.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-03-01T06:34:21.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2022-03-01T06:34:21.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2022-03-01T06:34:22.098 INFO:tasks.workunit.client.0.smithi169.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-03-01T06:34:22.106 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2022-03-01T06:34:22.107 INFO:tasks.workunit.client.0.smithi169.stdout:reading divergent objects 2022-03-01T06:34:22.107 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2022-03-01T06:34:22.412 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:34:22.413 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2022-03-01T06:34:03.546614+0000 44'102 48:150 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:32:28.568540+0000 0'0 2022-03-01T06:32:28.568540+0000 0 2022-03-01T06:34:22.413 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:34:22.413 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:34:22.414 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:34:22.423 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2022-03-01T06:34:22.424 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2022-03-01T06:34:22.425 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:34:22.425 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2022-03-01T06:34:22.462 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:34:22.462 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2022-03-01T06:34:22.491 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:34:22.492 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2022-03-01T06:34:22.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:34:22.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2022-03-01T06:34:22.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2022-03-01T06:34:22.551 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2022-03-01T06:34:22.552 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2022-03-01T06:34:22.555 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 26'101 (26'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2022-03-01T06:32:56.233621+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:34:22.555 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 20 _merge_object_divergent_entries: keeping 26'101 (26'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2022-03-01T06:32:56.233621+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:34:22.556 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-03-01T06:34:22.556 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 26'100 first_divergent_update: 26'101 last_divergent_update: 26'101 2022-03-01T06:34:22.556 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2022-03-01T06:34:22.556 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2022-03-01T06:34:22.557 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 26'101 2022-03-01T06:34:22.557 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 26'101 (26'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2022-03-01T06:32:56.233621+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:34:22.557 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:33:58.484+0000 7ff579e9e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2022-03-01T06:34:22.558 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2022-03-01T06:34:22.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2022-03-01T06:34:22.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2022-03-01T06:34:22.559 INFO:tasks.workunit.client.0.smithi169.stdout:success 2022-03-01T06:34:22.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2022-03-01T06:34:22.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2022-03-01T06:34:22.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2022-03-01T06:34:23.087 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'test' does not exist 2022-03-01T06:34:23.095 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2022-03-01T06:34:23.096 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:34:23.096 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:34:23.096 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:34:23.097 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:34:23.097 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:34:23.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:34:23.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2022-03-01T06:34:23.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-03-01T06:34:23.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:34:23.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-03-01T06:34:23.205 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:34:23.205 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:34:23.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:34:23.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:34:23.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:34:23.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:34:23.209 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:34:23.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:34:23.210 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:34:23.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:34:23.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:34:23.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:34:23.213 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:34:23.213 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:34:23.213 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:34:23.213 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:34:23.214 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:34:23.214 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:34:23.215 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-03-01T06:34:23.227 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:34:23.227 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:23.227 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:23.227 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31520 2022-03-01T06:34:23.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:34:23.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:34:23.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2022-03-01T06:34:23.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2022-03-01T06:34:23.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2022-03-01T06:34:23.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2022-03-01T06:34:23.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-03-01T06:34:23.231 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:34:23.231 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-03-01T06:34:23.231 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:34:23.231 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:34:23.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:34:23.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:34:23.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:34:23.234 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:34:23.234 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:34:23.234 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:34:23.234 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:34:23.235 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:34:23.235 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:34:23.235 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:34:23.235 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:34:23.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:34:23.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:34:23.236 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:34:23.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:34:23.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:34:23.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-03-01T06:34:23.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:34:23.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:23.239 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:23.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31520 2022-03-01T06:34:23.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:34:23.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:34:23.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2022-03-01T06:34:23.241 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-03-01T06:34:23.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:23.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:23.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.31520 2022-03-01T06:34:23.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-03-01T06:34:23.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2022-03-01T06:34:23.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-03-01T06:34:23.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2022-03-01T06:34:23.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2022-03-01T06:34:23.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2022-03-01T06:34:23.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2022-03-01T06:34:23.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2022-03-01T06:34:23.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2022-03-01T06:34:23.245 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2022-03-01T06:34:23.245 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2022-03-01T06:34:23.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2022-03-01T06:34:23.246 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:34:23.247 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:34:23.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2022-03-01T06:34:23.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2022-03-01T06:34:23.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-03-01T06:34:23.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-03-01T06:34:23.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-03-01T06:34:23.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2022-03-01T06:34:23.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2022-03-01T06:34:23.558 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-03-01T06:34:23.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:34:23.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:34:23.559 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:34:23.559 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:23.560 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:23.560 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:23.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-03-01T06:34:23.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-03-01T06:34:23.719 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-03-01T06:34:23.719 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:34:23.719 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:34:23.720 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-03-01T06:34:23.720 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-03-01T06:34:23.720 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:34:23.720 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:34:23.721 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:34:23.721 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:34:23.721 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:23.721 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:23.721 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-mon.a.asok 2022-03-01T06:34:23.722 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:34:23.722 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.31520/ceph-mon.a.asok config get fsid 2022-03-01T06:34:23.803 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-03-01T06:34:23.804 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:34:23.804 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:34:23.804 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-03-01T06:34:23.804 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-03-01T06:34:23.805 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:34:23.805 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:34:23.805 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:34:23.805 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:34:23.806 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:23.806 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:23.806 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-mon.a.asok 2022-03-01T06:34:23.806 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:34:23.807 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.31520/ceph-mon.a.asok config get mon_host 2022-03-01T06:34:23.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2022-03-01T06:34:23.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2022-03-01T06:34:23.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-03-01T06:34:23.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-03-01T06:34:23.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-03-01T06:34:23.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2022-03-01T06:34:23.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-03-01T06:34:24.776 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-03-01T06:34:24.776 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:34:24.776 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:34:24.776 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:34:24.777 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:24.777 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:24.777 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:24.777 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:34:24.778 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:34:24.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2022-03-01T06:34:24.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2022-03-01T06:34:24.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:34:24.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:34:24.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-03-01T06:34:24.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:34:24.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2022-03-01T06:34:24.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:34:24.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:34:24.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:34:24.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:34:24.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-03-01T06:34:24.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-03-01T06:34:24.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:34:24.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:34:24.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:34:24.809 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:34:24.810 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:34:24.810 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:34:24.810 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:34:24.810 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:24.810 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:24.811 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:24.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:24.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:34:24.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:34:24.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:34:24.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:34:24.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:34:24.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:34:24.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:34:24.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:24.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:34:24.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:34:24.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2022-03-01T06:34:24.814 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:34:24.814 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 2022-03-01T06:34:24.814 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8' 2022-03-01T06:34:24.814 INFO:tasks.workunit.client.0.smithi169.stdout:add osd0 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 2022-03-01T06:34:24.815 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:34:24.825 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBwvh1isBteMhAA4qRjmQx/z2LdJXQVnOhD+A== 2022-03-01T06:34:24.825 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBwvh1isBteMhAA4qRjmQx/z2LdJXQVnOhD+A=="}' 2022-03-01T06:34:24.825 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 -i td/divergent-priors/0/new.json 2022-03-01T06:34:25.112 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:34:25.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2022-03-01T06:34:25.122 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBwvh1isBteMhAA4qRjmQx/z2LdJXQVnOhD+A== --osd-uuid 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 2022-03-01T06:34:25.151 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.171+0000 7f8c1fcd5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:25.153 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.173+0000 7f8c1fcd5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:25.155 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.174+0000 7f8c1fcd5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:25.156 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.174+0000 7f8c1fcd5f00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:34:25.156 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.174+0000 7f8c1fcd5f00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:34:25.157 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.174+0000 7f8c1fcd5f00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:34:25.199 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.218+0000 7f8c1fcd5f00 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2022-03-01T06:34:25.593 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:25.612+0000 7f8c1fcd5f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:34:28.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2022-03-01T06:34:28.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:34:28.211 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd0 key to auth repository 2022-03-01T06:34:28.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-03-01T06:34:28.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:34:28.647 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-03-01T06:34:28.648 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:34:28.648 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:28.649 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:34:28.649 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:34:28.649 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:34:28.673 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:28.693+0000 7fb5e5249f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:28.675 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:28.695+0000 7fb5e5249f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:28.676 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:28.696+0000 7fb5e5249f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:28.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-03-01T06:34:28.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:34:28.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:34:28.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:34:28.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:34:28.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:28.984 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:34:28.984 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:34:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:28.985 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:34:29.193 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:29.213+0000 7fb5e5249f00 -1 Falling back to public interface 2022-03-01T06:34:29.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:34:29.906 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:29.926+0000 7fb5e5249f00 -1 osd.0 0 log_to_monitors {default=true} 2022-03-01T06:34:30.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:34:30.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:30.310 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:34:30.310 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:34:30.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:30.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:34:30.711 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/39595,v1:127.0.0.1:6803/39595] [v2:127.0.0.1:6804/39595,v1:127.0.0.1:6805/39595] exists,up 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 2022-03-01T06:34:30.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:34:30.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:34:30.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:34:30.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2022-03-01T06:34:30.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2022-03-01T06:34:30.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:34:30.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:34:30.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-03-01T06:34:30.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:34:30.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:34:30.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:34:30.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:34:30.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:34:30.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:34:30.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:34:30.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:34:30.723 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:34:30.723 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:34:30.723 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:34:30.723 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:34:30.723 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:34:30.724 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:34:30.724 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:34:30.724 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:30.724 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:30.725 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:30.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:30.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:34:30.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:34:30.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:34:30.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:34:30.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:34:30.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:34:30.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:34:30.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:30.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:34:30.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:34:30.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:34:30.727 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:34:30.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:34:30.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 37a8e729-ccba-40e7-90ef-c122436009c4' 2022-03-01T06:34:30.728 INFO:tasks.workunit.client.0.smithi169.stdout:add osd1 37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:34:30.729 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:34:30.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB2vh1iZHcALRAAePapYGKola6us0UfSn/5PQ== 2022-03-01T06:34:30.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB2vh1iZHcALRAAePapYGKola6us0UfSn/5PQ=="}' 2022-03-01T06:34:30.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 37a8e729-ccba-40e7-90ef-c122436009c4 -i td/divergent-priors/1/new.json 2022-03-01T06:34:31.240 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:34:31.251 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2022-03-01T06:34:31.252 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB2vh1iZHcALRAAePapYGKola6us0UfSn/5PQ== --osd-uuid 37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:34:31.281 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.301+0000 7fc866492f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:31.283 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.303+0000 7fc866492f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:31.283 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.303+0000 7fc866492f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:31.284 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.304+0000 7fc866492f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:34:31.285 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.304+0000 7fc866492f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:34:31.285 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.304+0000 7fc866492f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:34:31.331 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.351+0000 7fc866492f00 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2022-03-01T06:34:31.742 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:31.762+0000 7fc866492f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:34:34.267 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2022-03-01T06:34:34.268 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:34:34.269 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd1 key to auth repository 2022-03-01T06:34:34.270 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-03-01T06:34:34.270 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:34:34.703 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-03-01T06:34:34.704 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:34:34.705 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:34.705 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:34:34.705 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:34:34.705 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:34:34.731 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:34.751+0000 7fd525d44f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:34.733 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:34.753+0000 7fd525d44f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:34.734 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:34.753+0000 7fd525d44f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:35.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-03-01T06:34:35.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:34:35.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:34:35.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:34:35.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:34:35.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:35.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:34:35.061 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:34:35.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:35.062 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:34:35.249 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:35.269+0000 7fd525d44f00 -1 Falling back to public interface 2022-03-01T06:34:35.383 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:34:35.947 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:35.967+0000 7fd525d44f00 -1 osd.1 0 log_to_monitors {default=true} 2022-03-01T06:34:36.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:34:36.385 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:36.385 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:34:36.386 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:34:36.386 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:36.387 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:34:36.718 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/39958,v1:127.0.0.1:6811/39958] [v2:127.0.0.1:6812/39958,v1:127.0.0.1:6813/39958] exists,up 37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:34:36.724 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:34:36.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:34:36.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:34:36.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2022-03-01T06:34:36.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2022-03-01T06:34:36.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:34:36.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:34:36.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-03-01T06:34:36.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:34:36.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2022-03-01T06:34:36.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:34:36.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:34:36.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:34:36.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:34:36.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-03-01T06:34:36.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-03-01T06:34:36.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:34:36.730 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:34:36.730 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:34:36.730 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:34:36.730 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:34:36.731 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:34:36.731 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:34:36.731 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:34:36.732 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:34:36.732 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:36.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:34:36.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:34:36.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:34:36.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:34:36.734 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:34:36.734 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:34:36.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:34:36.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:34:36.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:36.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:34:36.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:34:36.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2022-03-01T06:34:36.736 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:34:36.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dfb1e4b2-84de-4539-b25a-a5ab79b41e6d 2022-03-01T06:34:36.737 INFO:tasks.workunit.client.0.smithi169.stdout:add osd2 dfb1e4b2-84de-4539-b25a-a5ab79b41e6d 2022-03-01T06:34:36.738 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 dfb1e4b2-84de-4539-b25a-a5ab79b41e6d' 2022-03-01T06:34:36.738 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:34:36.740 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB8vh1i8jBZLRAA7UfJrj+fIVX6y4XORSPwEQ== 2022-03-01T06:34:36.741 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB8vh1i8jBZLRAA7UfJrj+fIVX6y4XORSPwEQ=="}' 2022-03-01T06:34:36.741 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dfb1e4b2-84de-4539-b25a-a5ab79b41e6d -i td/divergent-priors/2/new.json 2022-03-01T06:34:37.173 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:34:37.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2022-03-01T06:34:37.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB8vh1i8jBZLRAA7UfJrj+fIVX6y4XORSPwEQ== --osd-uuid dfb1e4b2-84de-4539-b25a-a5ab79b41e6d 2022-03-01T06:34:37.213 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.233+0000 7f850a498f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:37.215 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.234+0000 7f850a498f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:37.215 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.235+0000 7f850a498f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:37.216 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.236+0000 7f850a498f00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:34:37.216 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.236+0000 7f850a498f00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:34:37.217 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.236+0000 7f850a498f00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:34:37.247 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.266+0000 7f850a498f00 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2022-03-01T06:34:37.666 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:37.686+0000 7f850a498f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:34:40.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2022-03-01T06:34:40.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:34:40.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-03-01T06:34:40.202 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd2 key to auth repository 2022-03-01T06:34:40.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:34:40.635 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:34:40.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-03-01T06:34:40.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:34:40.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:34:40.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:34:40.637 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:34:40.662 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:40.682+0000 7fc405210f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:40.664 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:40.684+0000 7fc405210f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:40.665 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:40.685+0000 7fc405210f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:34:40.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-03-01T06:34:40.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:34:40.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:34:40.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:34:40.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:34:40.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:40.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:34:40.998 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:34:40.998 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:40.999 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:34:41.182 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:41.202+0000 7fc405210f00 -1 Falling back to public interface 2022-03-01T06:34:41.319 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:34:41.930 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:34:41.949+0000 7fc405210f00 -1 osd.2 0 log_to_monitors {default=true} 2022-03-01T06:34:42.322 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:34:42.322 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:34:42.323 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:34:42.323 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:34:42.324 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:34:42.324 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:34:42.672 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/40313,v1:127.0.0.1:6819/40313] [v2:127.0.0.1:6820/40313,v1:127.0.0.1:6821/40313] exists,up dfb1e4b2-84de-4539-b25a-a5ab79b41e6d 2022-03-01T06:34:42.673 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:34:42.673 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:34:42.674 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:34:42.674 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2022-03-01T06:34:43.266 INFO:tasks.workunit.client.0.smithi169.stderr:noout is set 2022-03-01T06:34:43.274 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2022-03-01T06:34:43.876 INFO:tasks.workunit.client.0.smithi169.stderr:noin is set 2022-03-01T06:34:43.884 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2022-03-01T06:34:44.529 INFO:tasks.workunit.client.0.smithi169.stderr:nodown is set 2022-03-01T06:34:44.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2022-03-01T06:34:44.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2022-03-01T06:34:45.048 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'test' already exists 2022-03-01T06:34:45.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-03-01T06:34:46.058 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2022-03-01T06:34:46.642 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 size to 3 2022-03-01T06:34:46.650 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2022-03-01T06:34:47.439 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 min_size to 2 2022-03-01T06:34:47.447 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2022-03-01T06:34:47.448 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:34:47.448 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:34:47.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:34:47.774 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:34:47.774 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:34:47.774 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:34:47.774 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:34:47.775 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:34:47.897 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836486 2022-03-01T06:34:47.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836486 2022-03-01T06:34:47.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486' 2022-03-01T06:34:47.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:34:47.898 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:34:48.059 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672966 2022-03-01T06:34:48.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672966 2022-03-01T06:34:48.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2022-03-01T06:34:48.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:34:48.060 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:34:48.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542147 2022-03-01T06:34:48.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542147 2022-03-01T06:34:48.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-60129542147' 2022-03-01T06:34:48.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:34:48.186 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2022-03-01T06:34:48.186 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:34:48.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:34:48.187 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2022-03-01T06:34:48.187 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:34:48.188 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836486 2022-03-01T06:34:48.188 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2022-03-01T06:34:48.188 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836486 2022-03-01T06:34:48.189 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:34:48.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836486 2022-03-01T06:34:48.513 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:34:48.513 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672966 2022-03-01T06:34:48.513 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:34:48.514 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:34:48.515 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2022-03-01T06:34:48.515 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:34:48.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672966 2022-03-01T06:34:48.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2022-03-01T06:34:48.516 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 42949672966 2022-03-01T06:34:48.517 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:34:48.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672966 2022-03-01T06:34:48.843 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:34:48.843 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542147 2022-03-01T06:34:48.843 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:34:48.844 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:34:48.844 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542147 2022-03-01T06:34:48.845 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:34:48.845 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542147 2022-03-01T06:34:48.845 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2022-03-01T06:34:48.846 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 60129542147 2022-03-01T06:34:48.847 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:34:49.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542145 -lt 60129542147 2022-03-01T06:34:49.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:34:50.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:34:50.187 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:34:50.567 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542148 -lt 60129542147 2022-03-01T06:34:50.567 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2022-03-01T06:34:50.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:34:50.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:34:50.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:34:50.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:34:50.569 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2022-03-01T06:34:50.569 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:34:50.569 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:34:50.569 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:34:50.570 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:34:50.570 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:34:50.658 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:34:50.658 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:34:50.659 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:34:50.659 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:34:50.659 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:34:50.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:34:50.990 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:34:50.990 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:34:50.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:34:50.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:34:50.991 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:34:51.112 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836488 2022-03-01T06:34:51.113 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836488 2022-03-01T06:34:51.113 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488' 2022-03-01T06:34:51.113 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:34:51.113 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:34:51.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2022-03-01T06:34:51.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2022-03-01T06:34:51.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672968' 2022-03-01T06:34:51.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:34:51.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:34:51.360 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542150 2022-03-01T06:34:51.360 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542150 2022-03-01T06:34:51.360 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672968 2-60129542150' 2022-03-01T06:34:51.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:34:51.363 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836488 2022-03-01T06:34:51.363 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:34:51.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:34:51.365 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2022-03-01T06:34:51.365 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:34:51.366 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836488 2022-03-01T06:34:51.366 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2022-03-01T06:34:51.366 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836488 2022-03-01T06:34:51.367 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:34:51.686 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836488 2022-03-01T06:34:51.687 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:34:52.687 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:34:52.688 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:34:53.043 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836488 2022-03-01T06:34:53.043 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:34:53.043 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2022-03-01T06:34:53.044 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:34:53.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:34:53.045 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2022-03-01T06:34:53.045 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:34:53.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2022-03-01T06:34:53.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2022-03-01T06:34:53.047 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 42949672968 2022-03-01T06:34:53.047 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:34:53.372 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2022-03-01T06:34:53.373 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:34:53.373 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542150 2022-03-01T06:34:53.373 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:34:53.374 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:34:53.375 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542150 2022-03-01T06:34:53.375 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:34:53.376 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542150 2022-03-01T06:34:53.376 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2022-03-01T06:34:53.376 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 60129542150 2022-03-01T06:34:53.377 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:34:53.702 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542150 -lt 60129542150 2022-03-01T06:34:53.702 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:34:53.703 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:34:53.703 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:34:54.134 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:34:54.135 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:34:54.135 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:34:54.136 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:34:54.136 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:34:54.136 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:34:54.137 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:34:54.137 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:34:54.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-03-01T06:34:54.449 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:34:54.449 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:34:54.450 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:34:54.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-03-01T06:34:54.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-03-01T06:34:54.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-03-01T06:34:54.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2022-03-01T06:34:54.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2022-03-01T06:34:55.173 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:34:55.182 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2022-03-01T06:34:55.183 INFO:tasks.workunit.client.0.smithi169.stdout:primary and soon to be divergent is 1 2022-03-01T06:34:55.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2022-03-01T06:34:55.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:34:55.486 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:34:55.487 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-03-01T06:34:47.097874+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:34:55.487 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:34:55.487 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:34:55.488 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:34:55.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2022-03-01T06:34:55.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2022-03-01T06:34:55.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2022-03-01T06:34:55.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2022-03-01T06:34:55.499 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2022-03-01T06:34:55.499 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2022-03-01T06:34:55.499 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2022-03-01T06:34:55.499 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2022-03-01T06:34:55.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2022-03-01T06:34:55.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2022-03-01T06:34:55.500 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2022-03-01T06:34:55.500 INFO:tasks.workunit.client.0.smithi169.stdout:writing initial objects 2022-03-01T06:34:55.501 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2022-03-01T06:34:55.501 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:55.502 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2022-03-01T06:34:55.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:55.730 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2022-03-01T06:34:55.846 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:55.846 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2022-03-01T06:34:55.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:55.989 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2022-03-01T06:34:56.116 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.116 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2022-03-01T06:34:56.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2022-03-01T06:34:56.356 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.357 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2022-03-01T06:34:56.457 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.457 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2022-03-01T06:34:56.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2022-03-01T06:34:56.659 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.659 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2022-03-01T06:34:56.784 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.784 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2022-03-01T06:34:56.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:56.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2022-03-01T06:34:57.044 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2022-03-01T06:34:57.153 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.154 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2022-03-01T06:34:57.261 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.262 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2022-03-01T06:34:57.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2022-03-01T06:34:57.488 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.488 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2022-03-01T06:34:57.605 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.605 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2022-03-01T06:34:57.714 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.715 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2022-03-01T06:34:57.823 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.823 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2022-03-01T06:34:57.932 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:57.932 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2022-03-01T06:34:58.066 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.066 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2022-03-01T06:34:58.166 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2022-03-01T06:34:58.310 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.310 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2022-03-01T06:34:58.418 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.419 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2022-03-01T06:34:58.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2022-03-01T06:34:58.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2022-03-01T06:34:58.761 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.762 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2022-03-01T06:34:58.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:58.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2022-03-01T06:34:59.010 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.010 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2022-03-01T06:34:59.122 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.122 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2022-03-01T06:34:59.250 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.251 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2022-03-01T06:34:59.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2022-03-01T06:34:59.482 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.483 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2022-03-01T06:34:59.600 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.601 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2022-03-01T06:34:59.700 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.700 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2022-03-01T06:34:59.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2022-03-01T06:34:59.960 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:34:59.960 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2022-03-01T06:35:00.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.128 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2022-03-01T06:35:00.253 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.253 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2022-03-01T06:35:00.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2022-03-01T06:35:00.504 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.504 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2022-03-01T06:35:00.613 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.613 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2022-03-01T06:35:00.738 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.739 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2022-03-01T06:35:00.848 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.849 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2022-03-01T06:35:00.973 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:00.974 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2022-03-01T06:35:01.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.082 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2022-03-01T06:35:01.199 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2022-03-01T06:35:01.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2022-03-01T06:35:01.418 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.419 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2022-03-01T06:35:01.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2022-03-01T06:35:01.635 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2022-03-01T06:35:01.778 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.778 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2022-03-01T06:35:01.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2022-03-01T06:35:01.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:01.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2022-03-01T06:35:02.113 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.113 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2022-03-01T06:35:02.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2022-03-01T06:35:02.339 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.340 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2022-03-01T06:35:02.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2022-03-01T06:35:02.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2022-03-01T06:35:02.691 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.691 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2022-03-01T06:35:02.800 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.800 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2022-03-01T06:35:02.926 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:02.926 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2022-03-01T06:35:03.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2022-03-01T06:35:03.235 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2022-03-01T06:35:03.345 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.345 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2022-03-01T06:35:03.453 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.454 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2022-03-01T06:35:03.562 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.563 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2022-03-01T06:35:03.688 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.688 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2022-03-01T06:35:03.797 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.797 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2022-03-01T06:35:03.909 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:03.910 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2022-03-01T06:35:04.040 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.040 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2022-03-01T06:35:04.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2022-03-01T06:35:04.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2022-03-01T06:35:04.409 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.410 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2022-03-01T06:35:04.509 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.510 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2022-03-01T06:35:04.623 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.623 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2022-03-01T06:35:04.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2022-03-01T06:35:04.849 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.850 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2022-03-01T06:35:04.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:04.983 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2022-03-01T06:35:05.084 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.084 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2022-03-01T06:35:05.192 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.193 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2022-03-01T06:35:05.293 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.293 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2022-03-01T06:35:05.436 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.436 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2022-03-01T06:35:05.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2022-03-01T06:35:05.662 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.662 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2022-03-01T06:35:05.771 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.771 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2022-03-01T06:35:05.930 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:05.931 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2022-03-01T06:35:06.122 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.123 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2022-03-01T06:35:06.231 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.231 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2022-03-01T06:35:06.340 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.340 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2022-03-01T06:35:06.440 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.441 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2022-03-01T06:35:06.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2022-03-01T06:35:06.650 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.650 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2022-03-01T06:35:06.767 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.767 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2022-03-01T06:35:06.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:06.869 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2022-03-01T06:35:07.011 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:07.011 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2022-03-01T06:35:07.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:07.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2022-03-01T06:35:07.228 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:07.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2022-03-01T06:35:07.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2022-03-01T06:35:07.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2022-03-01T06:35:07.472 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:35:07.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2022-03-01T06:35:07.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:35:07.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:35:07.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:35:07.474 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:35:07.474 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:35:07.474 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:35:07.474 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:35:07.475 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:35:07.475 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:35:07.475 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:35:07.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:35:07.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:35:07.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:35:07.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:35:07.529 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:35:07.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:35:07.852 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:35:07.852 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:35:07.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:35:07.853 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:35:07.853 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:35:07.975 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836494 2022-03-01T06:35:07.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836494 2022-03-01T06:35:07.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494' 2022-03-01T06:35:07.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:35:07.976 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:35:08.099 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2022-03-01T06:35:08.099 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2022-03-01T06:35:08.099 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672974' 2022-03-01T06:35:08.100 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:35:08.100 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:35:08.234 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542155 2022-03-01T06:35:08.234 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542155 2022-03-01T06:35:08.235 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-60129542155' 2022-03-01T06:35:08.235 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:35:08.235 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836494 2022-03-01T06:35:08.235 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:35:08.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:35:08.237 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836494 2022-03-01T06:35:08.237 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:35:08.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836494 2022-03-01T06:35:08.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2022-03-01T06:35:08.238 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836494 2022-03-01T06:35:08.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:35:08.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836494 2022-03-01T06:35:08.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:35:08.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2022-03-01T06:35:08.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:35:08.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:35:08.560 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2022-03-01T06:35:08.560 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:35:08.561 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 42949672974 2022-03-01T06:35:08.561 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2022-03-01T06:35:08.562 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2022-03-01T06:35:08.562 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:35:08.888 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2022-03-01T06:35:08.889 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:35:08.889 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542155 2022-03-01T06:35:08.889 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:35:08.890 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:35:08.891 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542155 2022-03-01T06:35:08.891 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:35:08.891 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542155 2022-03-01T06:35:08.891 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2022-03-01T06:35:08.892 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 60129542155 2022-03-01T06:35:08.892 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:35:09.225 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542153 -lt 60129542155 2022-03-01T06:35:09.225 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:35:10.226 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:35:10.227 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:35:10.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542156 -lt 60129542155 2022-03-01T06:35:10.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:35:10.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:10.560 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:10.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:35:10.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:10.983 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:10.983 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:10.983 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:10.983 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:10.983 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:10.984 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:11.296 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-03-01T06:35:11.296 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:11.297 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:11.297 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:11.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-03-01T06:35:11.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-03-01T06:35:11.719 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-03-01T06:35:11.719 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2022-03-01T06:35:11.719 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2022-03-01T06:35:11.720 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2022-03-01T06:35:11.720 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2022-03-01T06:35:11.720 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2022-03-01T06:35:12.044 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2022-03-01T06:35:12.045 INFO:tasks.workunit.client.0.smithi169.stdout:blackholing osds 0 2 2022-03-01T06:35:12.045 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2022-03-01T06:35:12.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:35:12.350 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:35:12.350 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2022-03-01T06:34:47.097874+0000 27'100 27:113 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:35:12.350 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:35:12.351 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:35:12.352 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:35:12.360 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:12.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2022-03-01T06:35:12.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-03-01T06:35:12.361 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-03-01T06:35:12.362 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:35:12.362 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:35:12.362 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:35:12.362 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.0.asok 2022-03-01T06:35:12.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2022-03-01T06:35:12.363 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.0.asok config set objectstore_blackhole 1 2022-03-01T06:35:12.443 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:35:12.444 INFO:tasks.workunit.client.0.smithi169.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-03-01T06:35:12.444 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:35:12.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:12.451 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2022-03-01T06:35:12.452 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-03-01T06:35:12.452 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-03-01T06:35:12.452 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:35:12.452 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:35:12.453 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:35:12.453 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.2.asok 2022-03-01T06:35:12.453 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2022-03-01T06:35:12.453 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.2.asok config set objectstore_blackhole 1 2022-03-01T06:35:12.533 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:35:12.533 INFO:tasks.workunit.client.0.smithi169.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-03-01T06:35:12.534 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:35:12.540 INFO:tasks.workunit.client.0.smithi169.stdout:create new divergent objects 2022-03-01T06:35:12.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2022-03-01T06:35:12.542 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2022-03-01T06:35:12.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-03-01T06:35:12.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-03-01T06:35:12.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2022-03-01T06:35:12.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2022-03-01T06:35:12.543 INFO:tasks.workunit.client.0.smithi169.stdout:writing divergent objects 2022-03-01T06:35:12.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2022-03-01T06:35:12.545 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2022-03-01T06:35:12.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2022-03-01T06:35:12.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2022-03-01T06:35:12.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2022-03-01T06:35:12.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2022-03-01T06:35:12.546 INFO:tasks.workunit.client.0.smithi169.stdout:remove divergent objects 2022-03-01T06:35:12.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2022-03-01T06:35:12.548 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2022-03-01T06:35:12.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-03-01T06:35:12.549 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2022-03-01T06:35:12.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2022-03-01T06:35:12.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-03-01T06:35:12.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2022-03-01T06:35:12.550 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2022-03-01T06:35:12.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2022-03-01T06:35:12.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2022-03-01T06:35:12.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2022-03-01T06:35:22.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2022-03-01T06:35:22.551 INFO:tasks.workunit.client.0.smithi169.stderr:rados: no process found 2022-03-01T06:35:22.551 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2022-03-01T06:35:22.552 INFO:tasks.workunit.client.0.smithi169.stdout:killing all the osds 2022-03-01T06:35:22.552 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:35:22.864 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:35:22.865 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2022-03-01T06:34:47.097874+0000 27'106 27:119 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:35:22.865 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:35:22.865 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:35:22.866 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:35:22.874 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2022-03-01T06:35:22.875 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:35:22.876 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:35:22.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:35:22.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:35:22.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:35:22.981 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:35:22.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2022-03-01T06:35:22.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2022-03-01T06:35:23.442 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already down. 2022-03-01T06:35:23.451 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2022-03-01T06:35:23.451 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2022-03-01T06:35:23.955 INFO:tasks.workunit.client.0.smithi169.stderr:osd.1 is already down. 2022-03-01T06:35:23.962 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2022-03-01T06:35:23.962 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2022-03-01T06:35:24.447 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already down. 2022-03-01T06:35:24.455 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:24.456 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2022-03-01T06:35:24.928 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already out. 2022-03-01T06:35:24.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:24.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2022-03-01T06:35:25.395 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already out. 2022-03-01T06:35:25.403 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2022-03-01T06:35:25.405 INFO:tasks.workunit.client.0.smithi169.stdout:bringing up non_divergent 0 2 2022-03-01T06:35:25.405 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:35:25.711 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:35:25.712 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2022-03-01T06:34:47.097874+0000 27'106 27:119 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:35:25.712 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:35:25.712 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:35:25.713 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:35:25.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:25.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2022-03-01T06:35:25.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:35:25.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:35:25.726 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2022-03-01T06:35:25.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:35:25.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2022-03-01T06:35:25.727 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:35:25.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:35:25.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:35:25.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:35:25.728 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-03-01T06:35:25.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-03-01T06:35:25.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:35:25.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:35:25.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:35:25.730 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:35:25.730 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:35:25.730 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:35:25.730 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:35:25.730 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:35:25.731 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:35:25.731 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:35:25.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:35:25.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:35:25.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:35:25.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:35:25.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:35:25.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:35:25.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:35:25.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:35:25.733 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:35:25.734 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2022-03-01T06:35:25.734 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:35:25.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2022-03-01T06:35:25.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:35:25.736 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2022-03-01T06:35:25.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2022-03-01T06:35:25.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:35:25.736 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:35:25.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:35:25.752 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:25.772+0000 7fe93bf47f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:35:25.753 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:25.774+0000 7fe93bf47f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:35:25.754 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:25.775+0000 7fe93bf47f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:35:26.053 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2022-03-01T06:35:26.053 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:35:26.054 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:35:26.054 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:35:26.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:35:26.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:35:26.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:35:26.056 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:35:26.057 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:35:26.057 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:35:26.275 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:26.295+0000 7fe93bf47f00 -1 Falling back to public interface 2022-03-01T06:35:26.514 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:35:27.197 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:27.217+0000 7fe93bf47f00 -1 osd.0 27 log_to_monitors {default=true} 2022-03-01T06:35:27.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:35:27.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:35:27.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:35:27.516 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:35:27.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:35:27.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:35:27.848 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/44358,v1:127.0.0.1:6803/44358] [v2:127.0.0.1:6804/44358,v1:127.0.0.1:6805/44358] exists,up 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 2022-03-01T06:35:27.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:35:27.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:35:27.853 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:35:27.853 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:27.853 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2022-03-01T06:35:27.853 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:35:27.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:35:27.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2022-03-01T06:35:27.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:35:27.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2022-03-01T06:35:27.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:35:27.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:35:27.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:35:27.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:35:27.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-03-01T06:35:27.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-03-01T06:35:27.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:35:27.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:35:27.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:35:27.857 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:35:27.857 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:35:27.857 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:35:27.857 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:35:27.857 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:35:27.858 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:35:27.858 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:35:27.858 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:35:27.858 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:35:27.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:35:27.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:35:27.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:35:27.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:35:27.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:35:27.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:35:27.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:35:27.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2022-03-01T06:35:27.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2022-03-01T06:35:27.861 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:35:27.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:35:27.862 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2022-03-01T06:35:27.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2022-03-01T06:35:27.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:35:27.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:35:27.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:35:27.879 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:27.899+0000 7f13b8597f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:35:27.880 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:27.901+0000 7f13b8597f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:35:27.881 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:27.902+0000 7f13b8597f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:35:28.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2022-03-01T06:35:28.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:35:28.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:35:28.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:35:28.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:35:28.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:35:28.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:35:28.187 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:35:28.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:35:28.188 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:35:28.398 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:28.418+0000 7f13b8597f00 -1 Falling back to public interface 2022-03-01T06:35:28.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:35:29.270 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:35:29.290+0000 7f13b8597f00 -1 osd.2 27 log_to_monitors {default=true} 2022-03-01T06:35:29.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:35:29.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:35:29.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:35:29.531 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:35:29.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:35:29.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:35:29.913 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/44552,v1:127.0.0.1:6811/44552] [v2:127.0.0.1:6812/44552,v1:127.0.0.1:6813/44552] exists,up dfb1e4b2-84de-4539-b25a-a5ab79b41e6d 2022-03-01T06:35:29.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:35:29.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:35:29.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:35:29.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:29.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2022-03-01T06:35:30.414 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already in. 2022-03-01T06:35:30.422 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:35:30.423 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2022-03-01T06:35:30.878 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already in. 2022-03-01T06:35:30.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:35:30.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2022-03-01T06:35:30.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:35:30.887 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:35:30.888 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:35:30.888 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:35:30.888 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:35:30.888 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:35:30.889 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:35:30.889 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:35:30.890 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:35:30.890 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:35:30.943 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:35:30.943 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:35:30.943 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:35:30.944 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:35:30.944 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:35:31.289 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:35:31.290 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:35:31.290 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:35:31.290 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:35:31.291 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:35:31.291 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:35:31.414 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=146028888067 2022-03-01T06:35:31.415 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 146028888067 2022-03-01T06:35:31.415 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-146028888067' 2022-03-01T06:35:31.416 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:35:31.416 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:35:31.501 INFO:tasks.workunit.client.0.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-03-01T06:35:31.505 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-03-01T06:35:31.505 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-03-01T06:35:31.506 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-03-01T06:35:31.506 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:35:31.507 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:35:31.628 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2022-03-01T06:35:31.628 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2022-03-01T06:35:31.629 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-146028888067 2-154618822659' 2022-03-01T06:35:31.629 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:35:31.629 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-146028888067 2022-03-01T06:35:31.629 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:35:31.631 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:35:31.631 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-146028888067 2022-03-01T06:35:31.631 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:35:31.631 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=146028888067 2022-03-01T06:35:31.632 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2022-03-01T06:35:31.632 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 146028888067 2022-03-01T06:35:31.633 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:35:31.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 146028888067 2022-03-01T06:35:31.952 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:35:32.952 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:35:32.953 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:35:33.289 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 146028888067 -lt 146028888067 2022-03-01T06:35:33.289 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:35:33.289 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822659 2022-03-01T06:35:33.290 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:35:33.290 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:35:33.291 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822659 2022-03-01T06:35:33.291 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:35:33.292 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2022-03-01T06:35:33.292 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2022-03-01T06:35:33.292 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 154618822659 2022-03-01T06:35:33.293 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:35:33.615 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822659 -lt 154618822659 2022-03-01T06:35:33.616 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:35:33.616 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:33.616 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:34.055 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:35:34.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:34.056 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:34.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:34.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:34.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:34.058 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:34.058 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:34.377 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:34.378 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:34.378 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:34.378 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:34.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:34.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2022-03-01T06:35:34.796 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2022-03-01T06:35:34.796 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2022-03-01T06:35:34.796 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:34.796 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2022-03-01T06:35:34.896 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:34.897 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:34.897 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:34.897 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:34.898 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:34.898 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:34.898 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:34.899 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:35.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:35.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:35.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:35.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:35.633 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:35.633 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:35.634 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:35.634 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:35.634 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:35.634 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:35.634 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:35.635 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:35.635 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:36.057 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:36.057 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:36.057 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2022-03-01T06:35:36.058 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:36.058 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2022-03-01T06:35:36.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:36.259 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:36.259 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:36.259 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:36.259 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:36.259 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:36.260 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:36.260 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:36.571 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:36.571 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:36.572 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:36.572 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:36.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:36.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:36.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:36.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:36.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:36.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:36.995 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:36.995 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:36.995 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:37.412 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:37.412 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:37.412 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2022-03-01T06:35:37.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:37.413 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2022-03-01T06:35:37.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:37.814 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:37.814 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:37.814 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:37.814 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:37.815 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:37.815 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:37.815 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:38.128 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:38.129 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:38.129 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:38.129 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:38.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:38.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:38.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:38.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:38.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:38.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:38.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:38.555 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:38.555 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:38.975 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:38.975 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:38.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2022-03-01T06:35:38.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:38.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2022-03-01T06:35:39.777 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:39.777 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:39.777 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:39.778 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:39.778 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:39.778 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:39.778 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:39.778 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:40.100 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:40.100 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:40.101 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:40.101 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:40.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:40.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:40.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:40.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:40.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:40.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:40.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:40.541 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:40.541 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:40.981 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:40.981 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:40.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2022-03-01T06:35:40.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:40.982 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2022-03-01T06:35:42.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:42.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:42.583 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:42.584 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:42.584 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:42.584 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:42.584 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:42.584 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:42.909 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:42.909 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:42.910 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:42.910 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:43.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:43.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:43.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:43.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:43.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:43.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:43.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:43.331 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:43.331 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:43.748 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:43.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:43.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2022-03-01T06:35:43.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:43.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2022-03-01T06:35:46.949 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:46.949 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:46.951 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:46.951 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:46.951 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:46.951 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:46.952 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:46.952 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:47.275 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:47.276 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:47.276 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:47.276 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:47.695 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:47.696 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:47.696 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:47.696 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:47.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:47.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:47.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:47.697 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:47.698 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:48.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:48.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:48.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2022-03-01T06:35:48.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:48.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2022-03-01T06:35:54.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:35:54.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:35:54.550 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:35:54.550 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:35:54.551 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:35:54.551 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:35:54.551 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:35:54.552 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:35:54.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:35:54.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:35:54.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:35:54.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:35:55.296 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:35:55.297 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:35:55.297 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:35:55.298 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:35:55.298 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:35:55.299 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:35:55.299 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:35:55.299 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:35:55.299 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:35:55.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:35:55.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:35:55.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2022-03-01T06:35:55.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:35:55.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2022-03-01T06:36:03.021 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:03.022 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:03.023 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:03.024 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:03.024 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:03.024 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:03.024 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:03.024 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:03.349 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:03.349 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:03.350 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:03.350 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:03.771 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:03.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:03.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:03.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:03.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:03.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:03.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:03.773 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:03.774 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:04.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:04.335 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:04.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2022-03-01T06:36:04.337 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2022-03-01T06:36:04.646 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:36:04.646 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_fingerprint": "4b101960-99f0-4d69-83c4-35dc2dcfcfdf", 2022-03-01T06:36:04.647 INFO:tasks.workunit.client.0.smithi169.stdout: "version": "15.2.15-249-g45ece157", 2022-03-01T06:36:04.647 INFO:tasks.workunit.client.0.smithi169.stdout: "commit": "45ece157fc29b55a5ee952c81c0c23abdb89bf98", 2022-03-01T06:36:04.647 INFO:tasks.workunit.client.0.smithi169.stdout: "timestamp": "2022-03-01T06:36:04.666009+0000", 2022-03-01T06:36:04.648 INFO:tasks.workunit.client.0.smithi169.stdout: "tag": "", 2022-03-01T06:36:04.648 INFO:tasks.workunit.client.0.smithi169.stdout: "health": { 2022-03-01T06:36:04.648 INFO:tasks.workunit.client.0.smithi169.stdout: "status": "HEALTH_WARN", 2022-03-01T06:36:04.649 INFO:tasks.workunit.client.0.smithi169.stdout: "checks": { 2022-03-01T06:36:04.649 INFO:tasks.workunit.client.0.smithi169.stdout: "OSDMAP_FLAGS": { 2022-03-01T06:36:04.649 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:04.650 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:04.650 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "nodown,noout,noin flag(s) set", 2022-03-01T06:36:04.650 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 17 2022-03-01T06:36:04.650 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.651 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [], 2022-03-01T06:36:04.651 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:04.651 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.651 INFO:tasks.workunit.client.0.smithi169.stdout: "OSD_DOWN": { 2022-03-01T06:36:04.652 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:04.652 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:04.652 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "1 osds down", 2022-03-01T06:36:04.653 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:36:04.653 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.653 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:36:04.654 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.654 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "osd.1 (root=default,host=smithi169) is down" 2022-03-01T06:36:04.654 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.655 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.655 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:04.655 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.655 INFO:tasks.workunit.client.0.smithi169.stdout: "PG_DEGRADED": { 2022-03-01T06:36:04.656 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:04.656 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:04.656 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2022-03-01T06:36:04.656 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:36:04.657 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.657 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:36:04.657 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.657 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2022-03-01T06:36:04.658 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.658 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.658 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:04.658 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.659 INFO:tasks.workunit.client.0.smithi169.stdout: "POOL_APP_NOT_ENABLED": { 2022-03-01T06:36:04.659 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:04.659 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:04.659 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "1 pool(s) do not have an application enabled", 2022-03-01T06:36:04.660 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:36:04.660 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.660 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:36:04.660 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.661 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "application not enabled on pool 'test'" 2022-03-01T06:36:04.661 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.661 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.661 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2022-03-01T06:36:04.662 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.662 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.662 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:04.662 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.663 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.663 INFO:tasks.workunit.client.0.smithi169.stdout: "mutes": [] 2022-03-01T06:36:04.663 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.663 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap_first_committed": 1, 2022-03-01T06:36:04.664 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap_last_committed": 1, 2022-03-01T06:36:04.664 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap": { 2022-03-01T06:36:04.664 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 1, 2022-03-01T06:36:04.664 INFO:tasks.workunit.client.0.smithi169.stdout: "fsid": "03da9435-ccbd-45e1-be8b-88e9f3ce57a1", 2022-03-01T06:36:04.665 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:34:23.285055Z", 2022-03-01T06:36:04.665 INFO:tasks.workunit.client.0.smithi169.stdout: "created": "2022-03-01T06:34:23.285055Z", 2022-03-01T06:36:04.665 INFO:tasks.workunit.client.0.smithi169.stdout: "min_mon_release": 15, 2022-03-01T06:36:04.665 INFO:tasks.workunit.client.0.smithi169.stdout: "min_mon_release_name": "octopus", 2022-03-01T06:36:04.666 INFO:tasks.workunit.client.0.smithi169.stdout: "features": { 2022-03-01T06:36:04.666 INFO:tasks.workunit.client.0.smithi169.stdout: "persistent": [ 2022-03-01T06:36:04.666 INFO:tasks.workunit.client.0.smithi169.stdout: "kraken", 2022-03-01T06:36:04.666 INFO:tasks.workunit.client.0.smithi169.stdout: "luminous", 2022-03-01T06:36:04.667 INFO:tasks.workunit.client.0.smithi169.stdout: "mimic", 2022-03-01T06:36:04.667 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap-prune", 2022-03-01T06:36:04.667 INFO:tasks.workunit.client.0.smithi169.stdout: "nautilus", 2022-03-01T06:36:04.667 INFO:tasks.workunit.client.0.smithi169.stdout: "octopus" 2022-03-01T06:36:04.668 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.668 INFO:tasks.workunit.client.0.smithi169.stdout: "optional": [] 2022-03-01T06:36:04.668 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.668 INFO:tasks.workunit.client.0.smithi169.stdout: "mons": [ 2022-03-01T06:36:04.669 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.669 INFO:tasks.workunit.client.0.smithi169.stdout: "rank": 0, 2022-03-01T06:36:04.669 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "a", 2022-03-01T06:36:04.669 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:04.670 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.670 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.670 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.671 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:7115", 2022-03-01T06:36:04.671 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 0 2022-03-01T06:36:04.671 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.671 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.672 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.672 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:7115/0", 2022-03-01T06:36:04.672 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:7115/0", 2022-03-01T06:36:04.672 INFO:tasks.workunit.client.0.smithi169.stdout: "priority": 0, 2022-03-01T06:36:04.673 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 0 2022-03-01T06:36:04.673 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.673 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.674 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.674 INFO:tasks.workunit.client.0.smithi169.stdout: "quorum": [ 2022-03-01T06:36:04.674 INFO:tasks.workunit.client.0.smithi169.stdout: 0 2022-03-01T06:36:04.674 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.675 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap": { 2022-03-01T06:36:04.675 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 39, 2022-03-01T06:36:04.675 INFO:tasks.workunit.client.0.smithi169.stdout: "fsid": "03da9435-ccbd-45e1-be8b-88e9f3ce57a1", 2022-03-01T06:36:04.675 INFO:tasks.workunit.client.0.smithi169.stdout: "created": "2022-03-01T06:34:24.009894+0000", 2022-03-01T06:36:04.676 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:35:31.043500+0000", 2022-03-01T06:36:04.676 INFO:tasks.workunit.client.0.smithi169.stdout: "last_up_change": "2022-03-01T06:35:29.579208+0000", 2022-03-01T06:36:04.676 INFO:tasks.workunit.client.0.smithi169.stdout: "last_in_change": "2022-03-01T06:35:30.717028+0000", 2022-03-01T06:36:04.677 INFO:tasks.workunit.client.0.smithi169.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-03-01T06:36:04.677 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_num": 5800384, 2022-03-01T06:36:04.677 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_set": [ 2022-03-01T06:36:04.678 INFO:tasks.workunit.client.0.smithi169.stdout: "nodown", 2022-03-01T06:36:04.678 INFO:tasks.workunit.client.0.smithi169.stdout: "noin", 2022-03-01T06:36:04.678 INFO:tasks.workunit.client.0.smithi169.stdout: "noout", 2022-03-01T06:36:04.678 INFO:tasks.workunit.client.0.smithi169.stdout: "pglog_hardlimit", 2022-03-01T06:36:04.679 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snapdirs", 2022-03-01T06:36:04.679 INFO:tasks.workunit.client.0.smithi169.stdout: "recovery_deletes", 2022-03-01T06:36:04.679 INFO:tasks.workunit.client.0.smithi169.stdout: "sortbitwise" 2022-03-01T06:36:04.680 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.680 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_version": 9, 2022-03-01T06:36:04.680 INFO:tasks.workunit.client.0.smithi169.stdout: "full_ratio": 0.99000000953674316, 2022-03-01T06:36:04.680 INFO:tasks.workunit.client.0.smithi169.stdout: "backfillfull_ratio": 0.99000000953674316, 2022-03-01T06:36:04.681 INFO:tasks.workunit.client.0.smithi169.stdout: "nearfull_ratio": 0.85000002384185791, 2022-03-01T06:36:04.681 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_snapshot": "", 2022-03-01T06:36:04.681 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_max": 1, 2022-03-01T06:36:04.681 INFO:tasks.workunit.client.0.smithi169.stdout: "max_osd": 3, 2022-03-01T06:36:04.682 INFO:tasks.workunit.client.0.smithi169.stdout: "require_min_compat_client": "jewel", 2022-03-01T06:36:04.682 INFO:tasks.workunit.client.0.smithi169.stdout: "min_compat_client": "jewel", 2022-03-01T06:36:04.682 INFO:tasks.workunit.client.0.smithi169.stdout: "require_osd_release": "octopus", 2022-03-01T06:36:04.682 INFO:tasks.workunit.client.0.smithi169.stdout: "pools": [ 2022-03-01T06:36:04.683 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.683 INFO:tasks.workunit.client.0.smithi169.stdout: "pool": 1, 2022-03-01T06:36:04.683 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_name": "test", 2022-03-01T06:36:04.684 INFO:tasks.workunit.client.0.smithi169.stdout: "create_time": "2022-03-01T06:34:44.802792+0000", 2022-03-01T06:36:04.684 INFO:tasks.workunit.client.0.smithi169.stdout: "flags": 1, 2022-03-01T06:36:04.684 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_names": "hashpspool", 2022-03-01T06:36:04.685 INFO:tasks.workunit.client.0.smithi169.stdout: "type": 1, 2022-03-01T06:36:04.685 INFO:tasks.workunit.client.0.smithi169.stdout: "size": 3, 2022-03-01T06:36:04.685 INFO:tasks.workunit.client.0.smithi169.stdout: "min_size": 2, 2022-03-01T06:36:04.686 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_rule": 0, 2022-03-01T06:36:04.686 INFO:tasks.workunit.client.0.smithi169.stdout: "object_hash": 2, 2022-03-01T06:36:04.686 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_autoscale_mode": "off", 2022-03-01T06:36:04.687 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num": 1, 2022-03-01T06:36:04.687 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_placement_num": 1, 2022-03-01T06:36:04.687 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_placement_num_target": 1, 2022-03-01T06:36:04.687 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num_target": 1, 2022-03-01T06:36:04.688 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num_pending": 1, 2022-03-01T06:36:04.688 INFO:tasks.workunit.client.0.smithi169.stdout: "last_pg_merge_meta": { 2022-03-01T06:36:04.688 INFO:tasks.workunit.client.0.smithi169.stdout: "source_pgid": "0.0", 2022-03-01T06:36:04.688 INFO:tasks.workunit.client.0.smithi169.stdout: "ready_epoch": 0, 2022-03-01T06:36:04.689 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_started": 0, 2022-03-01T06:36:04.689 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_clean": 0, 2022-03-01T06:36:04.689 INFO:tasks.workunit.client.0.smithi169.stdout: "source_version": "0'0", 2022-03-01T06:36:04.689 INFO:tasks.workunit.client.0.smithi169.stdout: "target_version": "0'0" 2022-03-01T06:36:04.689 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.690 INFO:tasks.workunit.client.0.smithi169.stdout: "last_change": "27", 2022-03-01T06:36:04.690 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend": "0", 2022-03-01T06:36:04.690 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2022-03-01T06:36:04.690 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2022-03-01T06:36:04.691 INFO:tasks.workunit.client.0.smithi169.stdout: "auid": 0, 2022-03-01T06:36:04.691 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_mode": "selfmanaged", 2022-03-01T06:36:04.691 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_seq": 0, 2022-03-01T06:36:04.691 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_epoch": 0, 2022-03-01T06:36:04.692 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_snaps": [], 2022-03-01T06:36:04.692 INFO:tasks.workunit.client.0.smithi169.stdout: "removed_snaps": "[]", 2022-03-01T06:36:04.692 INFO:tasks.workunit.client.0.smithi169.stdout: "quota_max_bytes": 0, 2022-03-01T06:36:04.692 INFO:tasks.workunit.client.0.smithi169.stdout: "quota_max_objects": 0, 2022-03-01T06:36:04.692 INFO:tasks.workunit.client.0.smithi169.stdout: "tiers": [], 2022-03-01T06:36:04.693 INFO:tasks.workunit.client.0.smithi169.stdout: "tier_of": -1, 2022-03-01T06:36:04.693 INFO:tasks.workunit.client.0.smithi169.stdout: "read_tier": -1, 2022-03-01T06:36:04.693 INFO:tasks.workunit.client.0.smithi169.stdout: "write_tier": -1, 2022-03-01T06:36:04.693 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_mode": "none", 2022-03-01T06:36:04.693 INFO:tasks.workunit.client.0.smithi169.stdout: "target_max_bytes": 0, 2022-03-01T06:36:04.694 INFO:tasks.workunit.client.0.smithi169.stdout: "target_max_objects": 0, 2022-03-01T06:36:04.694 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-03-01T06:36:04.694 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-03-01T06:36:04.694 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2022-03-01T06:36:04.695 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_min_flush_age": 0, 2022-03-01T06:36:04.695 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_min_evict_age": 0, 2022-03-01T06:36:04.695 INFO:tasks.workunit.client.0.smithi169.stdout: "erasure_code_profile": "", 2022-03-01T06:36:04.695 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_params": { 2022-03-01T06:36:04.696 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "none" 2022-03-01T06:36:04.696 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.696 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_period": 0, 2022-03-01T06:36:04.696 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_count": 0, 2022-03-01T06:36:04.696 INFO:tasks.workunit.client.0.smithi169.stdout: "use_gmt_hitset": true, 2022-03-01T06:36:04.697 INFO:tasks.workunit.client.0.smithi169.stdout: "min_read_recency_for_promote": 0, 2022-03-01T06:36:04.697 INFO:tasks.workunit.client.0.smithi169.stdout: "min_write_recency_for_promote": 0, 2022-03-01T06:36:04.697 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2022-03-01T06:36:04.697 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_search_last_n": 0, 2022-03-01T06:36:04.698 INFO:tasks.workunit.client.0.smithi169.stdout: "grade_table": [], 2022-03-01T06:36:04.698 INFO:tasks.workunit.client.0.smithi169.stdout: "stripe_width": 0, 2022-03-01T06:36:04.698 INFO:tasks.workunit.client.0.smithi169.stdout: "expected_num_objects": 0, 2022-03-01T06:36:04.698 INFO:tasks.workunit.client.0.smithi169.stdout: "fast_read": false, 2022-03-01T06:36:04.698 INFO:tasks.workunit.client.0.smithi169.stdout: "options": {}, 2022-03-01T06:36:04.699 INFO:tasks.workunit.client.0.smithi169.stdout: "application_metadata": {} 2022-03-01T06:36:04.699 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.699 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.699 INFO:tasks.workunit.client.0.smithi169.stdout: "osds": [ 2022-03-01T06:36:04.699 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.700 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:04.700 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8", 2022-03-01T06:36:04.700 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 1, 2022-03-01T06:36:04.700 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:36:04.701 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:36:04.701 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:36:04.701 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 5, 2022-03-01T06:36:04.701 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 27, 2022-03-01T06:36:04.702 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 34, 2022-03-01T06:36:04.702 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 38, 2022-03-01T06:36:04.702 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 28, 2022-03-01T06:36:04.702 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:36:04.702 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:04.703 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.703 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.703 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.703 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6802", 2022-03-01T06:36:04.704 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.704 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.704 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.704 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.705 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6803", 2022-03-01T06:36:04.705 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.705 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.705 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.705 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.706 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:36:04.706 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.706 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.706 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.707 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6804", 2022-03-01T06:36:04.707 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.707 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.707 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.707 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.708 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6805", 2022-03-01T06:36:04.708 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.708 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.708 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.709 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.709 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:36:04.709 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.709 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.710 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.710 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6808", 2022-03-01T06:36:04.710 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.710 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.711 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.711 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.711 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6809", 2022-03-01T06:36:04.711 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.711 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.712 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.712 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.712 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:36:04.712 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.713 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.713 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.713 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6806", 2022-03-01T06:36:04.713 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.713 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.714 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.714 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.714 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6807", 2022-03-01T06:36:04.714 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:04.715 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.715 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.715 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.715 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6803/44358", 2022-03-01T06:36:04.716 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6805/44358", 2022-03-01T06:36:04.716 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6809/44358", 2022-03-01T06:36:04.716 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6807/44358", 2022-03-01T06:36:04.716 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:36:04.717 INFO:tasks.workunit.client.0.smithi169.stdout: "exists", 2022-03-01T06:36:04.717 INFO:tasks.workunit.client.0.smithi169.stdout: "up" 2022-03-01T06:36:04.717 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.717 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.717 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.718 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:04.718 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "37a8e729-ccba-40e7-90ef-c122436009c4", 2022-03-01T06:36:04.718 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 0, 2022-03-01T06:36:04.718 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:36:04.719 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:36:04.719 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:36:04.719 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 0, 2022-03-01T06:36:04.719 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 0, 2022-03-01T06:36:04.720 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 10, 2022-03-01T06:36:04.720 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 21, 2022-03-01T06:36:04.720 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 29, 2022-03-01T06:36:04.720 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:36:04.720 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:04.721 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.721 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.721 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.721 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6810", 2022-03-01T06:36:04.722 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.722 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.722 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.722 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.723 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6811", 2022-03-01T06:36:04.723 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.723 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.723 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.723 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.724 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:36:04.724 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.724 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.724 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.725 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6812", 2022-03-01T06:36:04.725 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.725 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.725 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.725 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.726 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6813", 2022-03-01T06:36:04.726 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.726 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.726 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.727 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.727 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:36:04.727 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.727 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.728 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.728 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6816", 2022-03-01T06:36:04.728 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.728 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.728 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.729 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.729 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6817", 2022-03-01T06:36:04.729 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.729 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.730 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.730 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.730 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:36:04.730 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.731 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.731 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.731 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6814", 2022-03-01T06:36:04.731 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.731 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.732 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.732 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.732 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6815", 2022-03-01T06:36:04.732 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:04.733 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.733 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.733 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.733 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6811/39958", 2022-03-01T06:36:04.734 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6813/39958", 2022-03-01T06:36:04.734 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39958", 2022-03-01T06:36:04.734 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39958", 2022-03-01T06:36:04.734 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:36:04.735 INFO:tasks.workunit.client.0.smithi169.stdout: "exists" 2022-03-01T06:36:04.735 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.735 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.735 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.735 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:04.736 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "dfb1e4b2-84de-4539-b25a-a5ab79b41e6d", 2022-03-01T06:36:04.736 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 1, 2022-03-01T06:36:04.736 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:36:04.736 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:36:04.737 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:36:04.737 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 14, 2022-03-01T06:36:04.737 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 27, 2022-03-01T06:36:04.737 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 36, 2022-03-01T06:36:04.738 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 0, 2022-03-01T06:36:04.738 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 30, 2022-03-01T06:36:04.738 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:36:04.738 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:04.738 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.739 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.739 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.739 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6810", 2022-03-01T06:36:04.739 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.740 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.740 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.740 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.740 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6811", 2022-03-01T06:36:04.741 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.741 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.741 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.741 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.741 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:36:04.742 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.742 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.742 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.742 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6812", 2022-03-01T06:36:04.743 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.743 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.743 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.743 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.744 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6813", 2022-03-01T06:36:04.744 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.744 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.744 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.744 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.745 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:36:04.745 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.745 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.745 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.746 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6816", 2022-03-01T06:36:04.746 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.746 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.746 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.746 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.747 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6817", 2022-03-01T06:36:04.747 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.752 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.752 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.752 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.753 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:36:04.753 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:04.753 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.753 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:04.754 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6814", 2022-03-01T06:36:04.754 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.754 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.754 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.755 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:04.755 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6815", 2022-03-01T06:36:04.755 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:04.755 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.755 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.756 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.756 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6811/44552", 2022-03-01T06:36:04.756 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6813/44552", 2022-03-01T06:36:04.756 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6817/44552", 2022-03-01T06:36:04.757 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6815/44552", 2022-03-01T06:36:04.757 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:36:04.757 INFO:tasks.workunit.client.0.smithi169.stdout: "exists", 2022-03-01T06:36:04.757 INFO:tasks.workunit.client.0.smithi169.stdout: "up" 2022-03-01T06:36:04.758 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.758 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.758 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.758 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_xinfo": [ 2022-03-01T06:36:04.759 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.759 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:04.759 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:35:23.287456+0000", 2022-03-01T06:36:04.759 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:36:04.759 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:36:04.760 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:36:04.760 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:36:04.760 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:34:30.918532+0000", 2022-03-01T06:36:04.760 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:36:04.761 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.761 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.761 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:04.761 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:35:23.759955+0000", 2022-03-01T06:36:04.761 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:36:04.762 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:36:04.762 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:36:04.762 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:36:04.762 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:34:36.971000+0000", 2022-03-01T06:36:04.763 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:36:04.763 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.763 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.763 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:04.763 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:35:24.290824+0000", 2022-03-01T06:36:04.764 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:36:04.764 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:36:04.764 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:36:04.764 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:36:04.765 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:34:42.935271+0000", 2022-03-01T06:36:04.765 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:36:04.765 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.765 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.766 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_upmap": [], 2022-03-01T06:36:04.766 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_upmap_items": [], 2022-03-01T06:36:04.766 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_temp": [], 2022-03-01T06:36:04.766 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_temp": [], 2022-03-01T06:36:04.766 INFO:tasks.workunit.client.0.smithi169.stdout: "blacklist": {}, 2022-03-01T06:36:04.767 INFO:tasks.workunit.client.0.smithi169.stdout: "erasure_code_profiles": { 2022-03-01T06:36:04.767 INFO:tasks.workunit.client.0.smithi169.stdout: "default": { 2022-03-01T06:36:04.767 INFO:tasks.workunit.client.0.smithi169.stdout: "k": "2", 2022-03-01T06:36:04.767 INFO:tasks.workunit.client.0.smithi169.stdout: "m": "2", 2022-03-01T06:36:04.768 INFO:tasks.workunit.client.0.smithi169.stdout: "plugin": "jerasure", 2022-03-01T06:36:04.768 INFO:tasks.workunit.client.0.smithi169.stdout: "technique": "reed_sol_van" 2022-03-01T06:36:04.768 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.768 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.768 INFO:tasks.workunit.client.0.smithi169.stdout: "removed_snaps_queue": [], 2022-03-01T06:36:04.769 INFO:tasks.workunit.client.0.smithi169.stdout: "new_removed_snaps": [], 2022-03-01T06:36:04.769 INFO:tasks.workunit.client.0.smithi169.stdout: "new_purged_snaps": [], 2022-03-01T06:36:04.769 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_node_flags": {}, 2022-03-01T06:36:04.769 INFO:tasks.workunit.client.0.smithi169.stdout: "device_class_flags": {} 2022-03-01T06:36:04.770 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.770 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_metadata": [ 2022-03-01T06:36:04.770 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.770 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:04.771 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:36:04.771 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6804/44358,v1:127.0.0.1:6805/44358]", 2022-03-01T06:36:04.771 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:36:04.771 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:36:04.771 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:36:04.772 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:36:04.772 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:36:04.772 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:36:04.772 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:36:04.773 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:36:04.773 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:36:04.773 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2022-03-01T06:36:04.773 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:36:04.773 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:36:04.774 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:36:04.774 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:36:04.774 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:36:04.774 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:36:04.775 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:36:04.775 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:36:04.775 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:36:04.775 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:36:04.775 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:36:04.776 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:36:04.776 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:36:04.776 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:36:04.776 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:36:04.777 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6802/44358,v1:127.0.0.1:6803/44358]", 2022-03-01T06:36:04.777 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:36:04.777 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/44358,v1:127.0.0.1:6809/44358]", 2022-03-01T06:36:04.777 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/44358,v1:127.0.0.1:6807/44358]", 2022-03-01T06:36:04.778 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:36:04.778 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:36:04.778 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:36:04.778 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:36:04.778 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:36:04.779 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:36:04.779 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:36:04.779 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:36:04.779 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:36:04.780 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/0", 2022-03-01T06:36:04.780 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:36:04.780 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:36:04.780 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:36:04.780 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.781 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.781 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:04.781 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:36:04.781 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6812/39958,v1:127.0.0.1:6813/39958]", 2022-03-01T06:36:04.782 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:36:04.782 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:36:04.782 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:36:04.782 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:36:04.783 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:36:04.783 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:36:04.783 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:36:04.783 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:36:04.784 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:36:04.784 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2022-03-01T06:36:04.784 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:36:04.784 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:36:04.785 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:36:04.785 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:36:04.785 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:36:04.785 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:36:04.785 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:36:04.786 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:36:04.786 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:36:04.786 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:36:04.786 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:36:04.787 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:36:04.787 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:36:04.787 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:36:04.787 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:36:04.788 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6810/39958,v1:127.0.0.1:6811/39958]", 2022-03-01T06:36:04.788 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:36:04.788 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39958,v1:127.0.0.1:6817/39958]", 2022-03-01T06:36:04.788 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39958,v1:127.0.0.1:6815/39958]", 2022-03-01T06:36:04.788 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:36:04.789 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:36:04.789 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:36:04.789 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:36:04.789 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:36:04.790 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:36:04.790 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:36:04.790 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:36:04.790 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:36:04.791 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/1", 2022-03-01T06:36:04.791 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:36:04.791 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:36:04.791 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:36:04.792 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.792 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.792 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:04.792 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:36:04.792 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6812/44552,v1:127.0.0.1:6813/44552]", 2022-03-01T06:36:04.793 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:36:04.793 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:36:04.793 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:36:04.793 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:36:04.794 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:36:04.794 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:36:04.794 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:36:04.794 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:36:04.795 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:36:04.795 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2022-03-01T06:36:04.795 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:36:04.795 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:36:04.795 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:36:04.796 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:36:04.796 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:36:04.796 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:36:04.796 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:36:04.797 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:36:04.797 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:36:04.797 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:36:04.797 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:36:04.797 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:36:04.798 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:36:04.798 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:36:04.798 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:36:04.799 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6810/44552,v1:127.0.0.1:6811/44552]", 2022-03-01T06:36:04.799 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:36:04.799 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/44552,v1:127.0.0.1:6817/44552]", 2022-03-01T06:36:04.799 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/44552,v1:127.0.0.1:6815/44552]", 2022-03-01T06:36:04.799 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:36:04.800 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:36:04.800 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:36:04.800 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:36:04.800 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:36:04.801 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:36:04.801 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:36:04.801 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:36:04.801 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:36:04.802 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/2", 2022-03-01T06:36:04.802 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:36:04.802 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:36:04.802 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:36:04.803 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.803 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.803 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_clean_epochs": { 2022-03-01T06:36:04.803 INFO:tasks.workunit.client.0.smithi169.stdout: "min_last_epoch_clean": 0, 2022-03-01T06:36:04.803 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_clean": { 2022-03-01T06:36:04.804 INFO:tasks.workunit.client.0.smithi169.stdout: "per_pool": [] 2022-03-01T06:36:04.804 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.804 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_epochs": [ 2022-03-01T06:36:04.804 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.805 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:04.805 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 34 2022-03-01T06:36:04.805 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.805 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.805 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:04.806 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 10 2022-03-01T06:36:04.806 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.806 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.806 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:04.807 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 36 2022-03-01T06:36:04.807 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.807 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.807 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.808 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_first_committed": 1, 2022-03-01T06:36:04.808 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_last_committed": 39, 2022-03-01T06:36:04.808 INFO:tasks.workunit.client.0.smithi169.stdout: "crushmap": { 2022-03-01T06:36:04.808 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": [ 2022-03-01T06:36:04.809 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.809 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:04.809 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.0", 2022-03-01T06:36:04.809 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:36:04.810 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.810 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.810 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:04.810 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.1", 2022-03-01T06:36:04.811 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:36:04.811 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.811 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.811 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:04.812 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.2", 2022-03-01T06:36:04.812 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:36:04.812 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.812 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.812 INFO:tasks.workunit.client.0.smithi169.stdout: "types": [ 2022-03-01T06:36:04.813 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.813 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 0, 2022-03-01T06:36:04.813 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd" 2022-03-01T06:36:04.813 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.814 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.814 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:36:04.814 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "host" 2022-03-01T06:36:04.814 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.814 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.815 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 2, 2022-03-01T06:36:04.815 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "chassis" 2022-03-01T06:36:04.815 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.815 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.816 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 3, 2022-03-01T06:36:04.816 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "rack" 2022-03-01T06:36:04.816 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.816 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.816 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 4, 2022-03-01T06:36:04.817 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "row" 2022-03-01T06:36:04.817 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.817 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.817 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 5, 2022-03-01T06:36:04.818 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "pdu" 2022-03-01T06:36:04.818 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.818 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.818 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 6, 2022-03-01T06:36:04.818 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "pod" 2022-03-01T06:36:04.819 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.819 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.819 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 7, 2022-03-01T06:36:04.819 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "room" 2022-03-01T06:36:04.820 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.820 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.820 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 8, 2022-03-01T06:36:04.820 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "datacenter" 2022-03-01T06:36:04.820 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.821 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.821 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 9, 2022-03-01T06:36:04.821 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "zone" 2022-03-01T06:36:04.821 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.822 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.822 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 10, 2022-03-01T06:36:04.822 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "region" 2022-03-01T06:36:04.822 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.823 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.823 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:36:04.823 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "root" 2022-03-01T06:36:04.823 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.823 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.824 INFO:tasks.workunit.client.0.smithi169.stdout: "buckets": [ 2022-03-01T06:36:04.824 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.824 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -1, 2022-03-01T06:36:04.824 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "default", 2022-03-01T06:36:04.825 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:36:04.825 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "root", 2022-03-01T06:36:04.825 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:04.825 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:04.825 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:04.826 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:04.826 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.826 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -3, 2022-03-01T06:36:04.826 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:04.827 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:04.827 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.827 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.827 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.828 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.828 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -2, 2022-03-01T06:36:04.828 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "default~hdd", 2022-03-01T06:36:04.828 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:36:04.828 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "root", 2022-03-01T06:36:04.829 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:04.829 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:04.829 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:04.829 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:04.830 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.830 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -4, 2022-03-01T06:36:04.830 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:04.830 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:04.830 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.831 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.831 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.831 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.831 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -3, 2022-03-01T06:36:04.832 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "smithi169", 2022-03-01T06:36:04.832 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:36:04.832 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "host", 2022-03-01T06:36:04.832 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:04.832 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:04.833 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:04.833 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:04.833 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.833 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:04.834 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:04.834 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:04.834 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.834 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.835 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:04.835 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:04.835 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 1 2022-03-01T06:36:04.835 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.835 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.836 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:04.836 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:04.836 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 2 2022-03-01T06:36:04.836 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.837 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.837 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.837 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.837 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -4, 2022-03-01T06:36:04.838 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "smithi169~hdd", 2022-03-01T06:36:04.838 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:36:04.838 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "host", 2022-03-01T06:36:04.838 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:04.839 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:04.839 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:04.839 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:04.839 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.839 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:04.840 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:04.840 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:04.840 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.840 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.841 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:04.841 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:04.841 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 1 2022-03-01T06:36:04.841 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.841 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.842 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:04.842 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:04.842 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 2 2022-03-01T06:36:04.842 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.843 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.843 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.843 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.843 INFO:tasks.workunit.client.0.smithi169.stdout: "rules": [ 2022-03-01T06:36:04.843 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.844 INFO:tasks.workunit.client.0.smithi169.stdout: "rule_id": 0, 2022-03-01T06:36:04.844 INFO:tasks.workunit.client.0.smithi169.stdout: "rule_name": "replicated_rule", 2022-03-01T06:36:04.844 INFO:tasks.workunit.client.0.smithi169.stdout: "ruleset": 0, 2022-03-01T06:36:04.844 INFO:tasks.workunit.client.0.smithi169.stdout: "type": 1, 2022-03-01T06:36:04.845 INFO:tasks.workunit.client.0.smithi169.stdout: "min_size": 1, 2022-03-01T06:36:04.845 INFO:tasks.workunit.client.0.smithi169.stdout: "max_size": 10, 2022-03-01T06:36:04.845 INFO:tasks.workunit.client.0.smithi169.stdout: "steps": [ 2022-03-01T06:36:04.845 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.845 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "take", 2022-03-01T06:36:04.846 INFO:tasks.workunit.client.0.smithi169.stdout: "item": -1, 2022-03-01T06:36:04.846 INFO:tasks.workunit.client.0.smithi169.stdout: "item_name": "default" 2022-03-01T06:36:04.846 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.846 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.847 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "choose_firstn", 2022-03-01T06:36:04.847 INFO:tasks.workunit.client.0.smithi169.stdout: "num": 0, 2022-03-01T06:36:04.847 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "osd" 2022-03-01T06:36:04.847 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.848 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.848 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "emit" 2022-03-01T06:36:04.848 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.848 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:04.848 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.849 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.849 INFO:tasks.workunit.client.0.smithi169.stdout: "tunables": { 2022-03-01T06:36:04.849 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_local_tries": 0, 2022-03-01T06:36:04.849 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_local_fallback_tries": 0, 2022-03-01T06:36:04.850 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_total_tries": 50, 2022-03-01T06:36:04.850 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_descend_once": 1, 2022-03-01T06:36:04.850 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_vary_r": 1, 2022-03-01T06:36:04.850 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_stable": 1, 2022-03-01T06:36:04.850 INFO:tasks.workunit.client.0.smithi169.stdout: "straw_calc_version": 1, 2022-03-01T06:36:04.851 INFO:tasks.workunit.client.0.smithi169.stdout: "allowed_bucket_algs": 54, 2022-03-01T06:36:04.851 INFO:tasks.workunit.client.0.smithi169.stdout: "profile": "jewel", 2022-03-01T06:36:04.851 INFO:tasks.workunit.client.0.smithi169.stdout: "optimal_tunables": 1, 2022-03-01T06:36:04.851 INFO:tasks.workunit.client.0.smithi169.stdout: "legacy_tunables": 0, 2022-03-01T06:36:04.852 INFO:tasks.workunit.client.0.smithi169.stdout: "minimum_required_version": "jewel", 2022-03-01T06:36:04.852 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables": 1, 2022-03-01T06:36:04.852 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables2": 1, 2022-03-01T06:36:04.852 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v2_rules": 0, 2022-03-01T06:36:04.852 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables3": 1, 2022-03-01T06:36:04.853 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v3_rules": 0, 2022-03-01T06:36:04.853 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v4_buckets": 1, 2022-03-01T06:36:04.853 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables5": 1, 2022-03-01T06:36:04.853 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v5_rules": 0 2022-03-01T06:36:04.854 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.854 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_args": {} 2022-03-01T06:36:04.854 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.854 INFO:tasks.workunit.client.0.smithi169.stdout: "fsmap": { 2022-03-01T06:36:04.854 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 1, 2022-03-01T06:36:04.855 INFO:tasks.workunit.client.0.smithi169.stdout: "default_fscid": -1, 2022-03-01T06:36:04.855 INFO:tasks.workunit.client.0.smithi169.stdout: "compat": { 2022-03-01T06:36:04.855 INFO:tasks.workunit.client.0.smithi169.stdout: "compat": {}, 2022-03-01T06:36:04.855 INFO:tasks.workunit.client.0.smithi169.stdout: "ro_compat": {}, 2022-03-01T06:36:04.856 INFO:tasks.workunit.client.0.smithi169.stdout: "incompat": { 2022-03-01T06:36:04.856 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_1": "base v0.20", 2022-03-01T06:36:04.856 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_2": "client writeable ranges", 2022-03-01T06:36:04.856 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_3": "default file layouts on dirs", 2022-03-01T06:36:04.856 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_4": "dir inode in separate object", 2022-03-01T06:36:04.857 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_5": "mds uses versioned encoding", 2022-03-01T06:36:04.857 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_6": "dirfrag is stored in omap", 2022-03-01T06:36:04.857 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_8": "no anchor table", 2022-03-01T06:36:04.857 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_9": "file layout v2", 2022-03-01T06:36:04.857 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_10": "snaprealm v2" 2022-03-01T06:36:04.858 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.858 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.858 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_flags": { 2022-03-01T06:36:04.858 INFO:tasks.workunit.client.0.smithi169.stdout: "enable_multiple": false, 2022-03-01T06:36:04.859 INFO:tasks.workunit.client.0.smithi169.stdout: "ever_enabled_multiple": false 2022-03-01T06:36:04.859 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.859 INFO:tasks.workunit.client.0.smithi169.stdout: "standbys": [], 2022-03-01T06:36:04.859 INFO:tasks.workunit.client.0.smithi169.stdout: "filesystems": [] 2022-03-01T06:36:04.860 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.860 INFO:tasks.workunit.client.0.smithi169.stdout: "mdsmap_first_committed": 1, 2022-03-01T06:36:04.860 INFO:tasks.workunit.client.0.smithi169.stdout: "mdsmap_last_committed": 1, 2022-03-01T06:36:04.860 INFO:tasks.workunit.client.0.smithi169.stdout: "auth": { 2022-03-01T06:36:04.860 INFO:tasks.workunit.client.0.smithi169.stdout: "first_committed": 1, 2022-03-01T06:36:04.861 INFO:tasks.workunit.client.0.smithi169.stdout: "last_committed": 5, 2022-03-01T06:36:04.861 INFO:tasks.workunit.client.0.smithi169.stdout: "num_secrets": 3 2022-03-01T06:36:04.861 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.861 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg": 1, 2022-03-01T06:36:04.862 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_active": 1, 2022-03-01T06:36:04.862 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_unknown": 0, 2022-03-01T06:36:04.862 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osd": 3, 2022-03-01T06:36:04.862 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_sum": { 2022-03-01T06:36:04.863 INFO:tasks.workunit.client.0.smithi169.stdout: "stat_sum": { 2022-03-01T06:36:04.863 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes": 48300, 2022-03-01T06:36:04.863 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects": 100, 2022-03-01T06:36:04.863 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_clones": 0, 2022-03-01T06:36:04.863 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_copies": 300, 2022-03-01T06:36:04.864 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing_on_primary": 0, 2022-03-01T06:36:04.864 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing": 0, 2022-03-01T06:36:04.864 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_degraded": 100, 2022-03-01T06:36:04.864 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_misplaced": 0, 2022-03-01T06:36:04.864 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_unfound": 0, 2022-03-01T06:36:04.865 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_dirty": 100, 2022-03-01T06:36:04.865 INFO:tasks.workunit.client.0.smithi169.stdout: "num_whiteouts": 0, 2022-03-01T06:36:04.865 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read": 0, 2022-03-01T06:36:04.865 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read_kb": 0, 2022-03-01T06:36:04.866 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write": 100, 2022-03-01T06:36:04.866 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write_kb": 100, 2022-03-01T06:36:04.866 INFO:tasks.workunit.client.0.smithi169.stdout: "num_scrub_errors": 0, 2022-03-01T06:36:04.866 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shallow_scrub_errors": 0, 2022-03-01T06:36:04.866 INFO:tasks.workunit.client.0.smithi169.stdout: "num_deep_scrub_errors": 0, 2022-03-01T06:36:04.867 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_recovered": 0, 2022-03-01T06:36:04.867 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_recovered": 0, 2022-03-01T06:36:04.867 INFO:tasks.workunit.client.0.smithi169.stdout: "num_keys_recovered": 0, 2022-03-01T06:36:04.867 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_omap": 0, 2022-03-01T06:36:04.868 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_hit_set_archive": 0, 2022-03-01T06:36:04.868 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_hit_set_archive": 0, 2022-03-01T06:36:04.868 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush": 0, 2022-03-01T06:36:04.868 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_kb": 0, 2022-03-01T06:36:04.868 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict": 0, 2022-03-01T06:36:04.869 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_kb": 0, 2022-03-01T06:36:04.869 INFO:tasks.workunit.client.0.smithi169.stdout: "num_promote": 0, 2022-03-01T06:36:04.869 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_high": 0, 2022-03-01T06:36:04.869 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_low": 0, 2022-03-01T06:36:04.870 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_some": 0, 2022-03-01T06:36:04.870 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_full": 0, 2022-03-01T06:36:04.870 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_pinned": 0, 2022-03-01T06:36:04.870 INFO:tasks.workunit.client.0.smithi169.stdout: "num_legacy_snapsets": 0, 2022-03-01T06:36:04.870 INFO:tasks.workunit.client.0.smithi169.stdout: "num_large_omap_objects": 0, 2022-03-01T06:36:04.871 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_manifest": 0, 2022-03-01T06:36:04.871 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_bytes": 0, 2022-03-01T06:36:04.871 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_keys": 0, 2022-03-01T06:36:04.871 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_repaired": 0 2022-03-01T06:36:04.872 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.872 INFO:tasks.workunit.client.0.smithi169.stdout: "store_stats": { 2022-03-01T06:36:04.872 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 0, 2022-03-01T06:36:04.872 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 0, 2022-03-01T06:36:04.872 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 0, 2022-03-01T06:36:04.873 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 0, 2022-03-01T06:36:04.873 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 0, 2022-03-01T06:36:04.873 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:04.873 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:04.874 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:04.874 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 0, 2022-03-01T06:36:04.874 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 0 2022-03-01T06:36:04.874 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.875 INFO:tasks.workunit.client.0.smithi169.stdout: "log_size": 16, 2022-03-01T06:36:04.875 INFO:tasks.workunit.client.0.smithi169.stdout: "ondisk_log_size": 16, 2022-03-01T06:36:04.875 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 2, 2022-03-01T06:36:04.875 INFO:tasks.workunit.client.0.smithi169.stdout: "acting": 2, 2022-03-01T06:36:04.875 INFO:tasks.workunit.client.0.smithi169.stdout: "num_store_stats": 0 2022-03-01T06:36:04.876 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.876 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_sum": { 2022-03-01T06:36:04.876 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 0, 2022-03-01T06:36:04.876 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 0, 2022-03-01T06:36:04.877 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pgs": 3, 2022-03-01T06:36:04.877 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osds": 3, 2022-03-01T06:36:04.877 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_osds": 3, 2022-03-01T06:36:04.877 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_omap_osds": 3, 2022-03-01T06:36:04.877 INFO:tasks.workunit.client.0.smithi169.stdout: "kb": 314572800, 2022-03-01T06:36:04.878 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used": 3178944, 2022-03-01T06:36:04.878 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_data": 32896, 2022-03-01T06:36:04.878 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_omap": 12, 2022-03-01T06:36:04.878 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_meta": 3145715, 2022-03-01T06:36:04.879 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_avail": 311393856, 2022-03-01T06:36:04.879 INFO:tasks.workunit.client.0.smithi169.stdout: "statfs": { 2022-03-01T06:36:04.879 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 322122547200, 2022-03-01T06:36:04.879 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 318867308544, 2022-03-01T06:36:04.880 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 3221225472, 2022-03-01T06:36:04.880 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 33685504, 2022-03-01T06:36:04.880 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 403752, 2022-03-01T06:36:04.880 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:04.880 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:04.881 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:04.881 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 13214, 2022-03-01T06:36:04.881 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 3221212258 2022-03-01T06:36:04.881 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.882 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_peers": [], 2022-03-01T06:36:04.882 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_trim_queue_len": 0, 2022-03-01T06:36:04.882 INFO:tasks.workunit.client.0.smithi169.stdout: "num_snap_trimming": 0, 2022-03-01T06:36:04.882 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shards_repaired": 0, 2022-03-01T06:36:04.882 INFO:tasks.workunit.client.0.smithi169.stdout: "op_queue_age_hist": { 2022-03-01T06:36:04.883 INFO:tasks.workunit.client.0.smithi169.stdout: "histogram": [], 2022-03-01T06:36:04.883 INFO:tasks.workunit.client.0.smithi169.stdout: "upper_bound": 1 2022-03-01T06:36:04.883 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.883 INFO:tasks.workunit.client.0.smithi169.stdout: "perf_stat": { 2022-03-01T06:36:04.884 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ms": 0, 2022-03-01T06:36:04.884 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ms": 0, 2022-03-01T06:36:04.884 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ns": 0, 2022-03-01T06:36:04.884 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ns": 0 2022-03-01T06:36:04.884 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.885 INFO:tasks.workunit.client.0.smithi169.stdout: "alerts": [], 2022-03-01T06:36:04.885 INFO:tasks.workunit.client.0.smithi169.stdout: "network_ping_times": [] 2022-03-01T06:36:04.885 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.885 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_sum_by_class": { 2022-03-01T06:36:04.886 INFO:tasks.workunit.client.0.smithi169.stdout: "hdd": { 2022-03-01T06:36:04.886 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 0, 2022-03-01T06:36:04.886 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 0, 2022-03-01T06:36:04.886 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pgs": 3, 2022-03-01T06:36:04.887 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osds": 3, 2022-03-01T06:36:04.887 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_osds": 3, 2022-03-01T06:36:04.887 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_omap_osds": 3, 2022-03-01T06:36:04.887 INFO:tasks.workunit.client.0.smithi169.stdout: "kb": 314572800, 2022-03-01T06:36:04.887 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used": 3178944, 2022-03-01T06:36:04.888 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_data": 32896, 2022-03-01T06:36:04.888 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_omap": 12, 2022-03-01T06:36:04.888 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_meta": 3145715, 2022-03-01T06:36:04.888 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_avail": 311393856, 2022-03-01T06:36:04.889 INFO:tasks.workunit.client.0.smithi169.stdout: "statfs": { 2022-03-01T06:36:04.889 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 322122547200, 2022-03-01T06:36:04.889 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 318867308544, 2022-03-01T06:36:04.889 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 3221225472, 2022-03-01T06:36:04.890 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 33685504, 2022-03-01T06:36:04.890 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 403752, 2022-03-01T06:36:04.890 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:04.890 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:04.891 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:04.891 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 13214, 2022-03-01T06:36:04.891 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 3221212258 2022-03-01T06:36:04.891 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.891 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_peers": [], 2022-03-01T06:36:04.892 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_trim_queue_len": 0, 2022-03-01T06:36:04.892 INFO:tasks.workunit.client.0.smithi169.stdout: "num_snap_trimming": 0, 2022-03-01T06:36:04.892 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shards_repaired": 0, 2022-03-01T06:36:04.892 INFO:tasks.workunit.client.0.smithi169.stdout: "op_queue_age_hist": { 2022-03-01T06:36:04.893 INFO:tasks.workunit.client.0.smithi169.stdout: "histogram": [], 2022-03-01T06:36:04.893 INFO:tasks.workunit.client.0.smithi169.stdout: "upper_bound": 1 2022-03-01T06:36:04.893 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.893 INFO:tasks.workunit.client.0.smithi169.stdout: "perf_stat": { 2022-03-01T06:36:04.893 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ms": 0, 2022-03-01T06:36:04.894 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ms": 0, 2022-03-01T06:36:04.894 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ns": 0, 2022-03-01T06:36:04.894 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ns": 0 2022-03-01T06:36:04.894 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.895 INFO:tasks.workunit.client.0.smithi169.stdout: "alerts": [], 2022-03-01T06:36:04.895 INFO:tasks.workunit.client.0.smithi169.stdout: "network_ping_times": [] 2022-03-01T06:36:04.895 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.895 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.895 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_stats": [ 2022-03-01T06:36:04.896 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.896 INFO:tasks.workunit.client.0.smithi169.stdout: "poolid": 1, 2022-03-01T06:36:04.896 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg": 1, 2022-03-01T06:36:04.896 INFO:tasks.workunit.client.0.smithi169.stdout: "stat_sum": { 2022-03-01T06:36:04.897 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes": 48300, 2022-03-01T06:36:04.897 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects": 100, 2022-03-01T06:36:04.897 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_clones": 0, 2022-03-01T06:36:04.897 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_copies": 300, 2022-03-01T06:36:04.897 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing_on_primary": 0, 2022-03-01T06:36:04.898 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing": 0, 2022-03-01T06:36:04.898 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_degraded": 100, 2022-03-01T06:36:04.898 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_misplaced": 0, 2022-03-01T06:36:04.898 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_unfound": 0, 2022-03-01T06:36:04.899 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_dirty": 100, 2022-03-01T06:36:04.899 INFO:tasks.workunit.client.0.smithi169.stdout: "num_whiteouts": 0, 2022-03-01T06:36:04.899 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read": 0, 2022-03-01T06:36:04.899 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read_kb": 0, 2022-03-01T06:36:04.899 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write": 100, 2022-03-01T06:36:04.900 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write_kb": 100, 2022-03-01T06:36:04.900 INFO:tasks.workunit.client.0.smithi169.stdout: "num_scrub_errors": 0, 2022-03-01T06:36:04.900 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shallow_scrub_errors": 0, 2022-03-01T06:36:04.900 INFO:tasks.workunit.client.0.smithi169.stdout: "num_deep_scrub_errors": 0, 2022-03-01T06:36:04.901 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_recovered": 0, 2022-03-01T06:36:04.901 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_recovered": 0, 2022-03-01T06:36:04.901 INFO:tasks.workunit.client.0.smithi169.stdout: "num_keys_recovered": 0, 2022-03-01T06:36:04.901 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_omap": 0, 2022-03-01T06:36:04.901 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_hit_set_archive": 0, 2022-03-01T06:36:04.902 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_hit_set_archive": 0, 2022-03-01T06:36:04.902 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush": 0, 2022-03-01T06:36:04.902 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_kb": 0, 2022-03-01T06:36:04.902 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict": 0, 2022-03-01T06:36:04.903 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_kb": 0, 2022-03-01T06:36:04.903 INFO:tasks.workunit.client.0.smithi169.stdout: "num_promote": 0, 2022-03-01T06:36:04.903 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_high": 0, 2022-03-01T06:36:04.903 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_low": 0, 2022-03-01T06:36:04.903 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_some": 0, 2022-03-01T06:36:04.904 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_full": 0, 2022-03-01T06:36:04.904 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_pinned": 0, 2022-03-01T06:36:04.904 INFO:tasks.workunit.client.0.smithi169.stdout: "num_legacy_snapsets": 0, 2022-03-01T06:36:04.904 INFO:tasks.workunit.client.0.smithi169.stdout: "num_large_omap_objects": 0, 2022-03-01T06:36:04.905 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_manifest": 0, 2022-03-01T06:36:04.905 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_bytes": 0, 2022-03-01T06:36:04.905 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_keys": 0, 2022-03-01T06:36:04.905 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_repaired": 0 2022-03-01T06:36:04.905 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.906 INFO:tasks.workunit.client.0.smithi169.stdout: "store_stats": { 2022-03-01T06:36:04.906 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 0, 2022-03-01T06:36:04.906 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 0, 2022-03-01T06:36:04.906 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 0, 2022-03-01T06:36:04.907 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 19529728, 2022-03-01T06:36:04.907 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 143228, 2022-03-01T06:36:04.907 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:04.907 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:04.908 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:04.908 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 0, 2022-03-01T06:36:04.908 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 0 2022-03-01T06:36:04.908 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.908 INFO:tasks.workunit.client.0.smithi169.stdout: "log_size": 16, 2022-03-01T06:36:04.909 INFO:tasks.workunit.client.0.smithi169.stdout: "ondisk_log_size": 16, 2022-03-01T06:36:04.909 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 2, 2022-03-01T06:36:04.909 INFO:tasks.workunit.client.0.smithi169.stdout: "acting": 2, 2022-03-01T06:36:04.909 INFO:tasks.workunit.client.0.smithi169.stdout: "num_store_stats": 3 2022-03-01T06:36:04.910 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.910 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.910 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_stats": [ 2022-03-01T06:36:04.910 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.910 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:04.911 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 146028888074 2022-03-01T06:36:04.911 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.911 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.911 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:04.912 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 42949672977 2022-03-01T06:36:04.912 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.912 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.912 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:04.913 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 154618822665 2022-03-01T06:36:04.913 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.913 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.913 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_by_state": [ 2022-03-01T06:36:04.913 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.914 INFO:tasks.workunit.client.0.smithi169.stdout: "state": "active+undersized+degraded", 2022-03-01T06:36:04.914 INFO:tasks.workunit.client.0.smithi169.stdout: "num": 1 2022-03-01T06:36:04.914 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.914 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.915 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_by_osd": [ 2022-03-01T06:36:04.915 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.915 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:04.915 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 0, 2022-03-01T06:36:04.915 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 1, 2022-03-01T06:36:04.916 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:36:04.916 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.916 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.916 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:04.917 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 0, 2022-03-01T06:36:04.917 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 0, 2022-03-01T06:36:04.917 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:36:04.917 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.918 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.918 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:04.918 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 1, 2022-03-01T06:36:04.918 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 1, 2022-03-01T06:36:04.918 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:36:04.919 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.919 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.919 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snaps": [ 2022-03-01T06:36:04.919 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:04.920 INFO:tasks.workunit.client.0.smithi169.stdout: "pool": 1, 2022-03-01T06:36:04.920 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snaps": {} 2022-03-01T06:36:04.920 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.920 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:04.920 INFO:tasks.workunit.client.0.smithi169.stdout: "servicemap": { 2022-03-01T06:36:04.921 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 3, 2022-03-01T06:36:04.921 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:35:38.142137+0000", 2022-03-01T06:36:04.921 INFO:tasks.workunit.client.0.smithi169.stdout: "services": { 2022-03-01T06:36:04.921 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": { 2022-03-01T06:36:04.922 INFO:tasks.workunit.client.0.smithi169.stdout: "daemons": { 2022-03-01T06:36:04.922 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": "", 2022-03-01T06:36:04.922 INFO:tasks.workunit.client.0.smithi169.stdout: "0": { 2022-03-01T06:36:04.922 INFO:tasks.workunit.client.0.smithi169.stdout: "start_epoch": 0, 2022-03-01T06:36:04.922 INFO:tasks.workunit.client.0.smithi169.stdout: "start_stamp": "0.000000", 2022-03-01T06:36:04.923 INFO:tasks.workunit.client.0.smithi169.stdout: "gid": 0, 2022-03-01T06:36:04.923 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "(unrecognized address family 0)/0", 2022-03-01T06:36:04.923 INFO:tasks.workunit.client.0.smithi169.stdout: "metadata": {}, 2022-03-01T06:36:04.923 INFO:tasks.workunit.client.0.smithi169.stdout: "task_status": {} 2022-03-01T06:36:04.924 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.924 INFO:tasks.workunit.client.0.smithi169.stdout: "2": { 2022-03-01T06:36:04.924 INFO:tasks.workunit.client.0.smithi169.stdout: "start_epoch": 0, 2022-03-01T06:36:04.924 INFO:tasks.workunit.client.0.smithi169.stdout: "start_stamp": "0.000000", 2022-03-01T06:36:04.924 INFO:tasks.workunit.client.0.smithi169.stdout: "gid": 0, 2022-03-01T06:36:04.925 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "(unrecognized address family 0)/0", 2022-03-01T06:36:04.925 INFO:tasks.workunit.client.0.smithi169.stdout: "metadata": {}, 2022-03-01T06:36:04.925 INFO:tasks.workunit.client.0.smithi169.stdout: "task_status": {} 2022-03-01T06:36:04.925 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.926 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.926 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.926 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.926 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:04.927 INFO:tasks.workunit.client.0.smithi169.stdout: "paxos": { 2022-03-01T06:36:04.927 INFO:tasks.workunit.client.0.smithi169.stdout: "first_committed": 1, 2022-03-01T06:36:04.927 INFO:tasks.workunit.client.0.smithi169.stdout: "last_committed": 330, 2022-03-01T06:36:04.927 INFO:tasks.workunit.client.0.smithi169.stdout: "last_pn": 0, 2022-03-01T06:36:04.928 INFO:tasks.workunit.client.0.smithi169.stdout: "accepted_pn": 0 2022-03-01T06:36:04.928 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:04.928 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:36:04.929 INFO:tasks.workunit.client.0.smithi169.stderr:report 180629939 2022-03-01T06:36:04.930 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2022-03-01T06:36:04.930 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2022-03-01T06:36:04.930 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2022-03-01T06:36:04.930 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2022-03-01T06:36:04.931 INFO:tasks.workunit.client.0.smithi169.stdout:writing non-divergent object existing_4 2022-03-01T06:36:04.931 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:04.965 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:04.966 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2022-03-01T06:35:31.320549+0000 27'100 39:121 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:04.966 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:04.966 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:04.967 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:04.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2022-03-01T06:36:05.062 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:36:05.063 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2022-03-01T06:36:05.063 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:36:05.063 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:36:05.063 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:36:05.064 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:36:05.064 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:36:05.064 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:36:05.064 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:36:05.065 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:36:05.065 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:36:05.065 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:36:05.118 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:36:05.118 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:36:05.118 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:36:05.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:36:05.119 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:36:05.452 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:36:05.452 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:36:05.453 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:36:05.453 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:36:05.453 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:36:05.453 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:36:05.576 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=146028888076 2022-03-01T06:36:05.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 146028888076 2022-03-01T06:36:05.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-146028888076' 2022-03-01T06:36:05.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:36:05.577 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:36:05.664 INFO:tasks.workunit.client.0.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-03-01T06:36:05.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-03-01T06:36:05.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-03-01T06:36:05.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-03-01T06:36:05.666 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:36:05.667 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:36:05.789 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822668 2022-03-01T06:36:05.790 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822668 2022-03-01T06:36:05.790 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-146028888076 2-154618822668' 2022-03-01T06:36:05.790 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:36:05.791 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-146028888076 2022-03-01T06:36:05.791 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:36:05.792 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:36:05.792 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-146028888076 2022-03-01T06:36:05.792 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:36:05.793 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=146028888076 2022-03-01T06:36:05.793 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 146028888076' 2022-03-01T06:36:05.793 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 146028888076 2022-03-01T06:36:05.794 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:36:06.137 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 146028888074 -lt 146028888076 2022-03-01T06:36:06.137 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:36:07.138 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:36:07.139 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:36:07.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 146028888076 -lt 146028888076 2022-03-01T06:36:07.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:36:07.473 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822668 2022-03-01T06:36:07.474 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:36:07.474 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:36:07.475 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822668 2022-03-01T06:36:07.475 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:36:07.476 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822668 2022-03-01T06:36:07.476 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2022-03-01T06:36:07.476 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 154618822668 2022-03-01T06:36:07.477 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:36:07.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822668 -lt 154618822668 2022-03-01T06:36:07.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:36:07.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:07.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:08.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:36:08.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:08.240 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:08.240 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:08.240 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:08.241 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:08.241 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:08.241 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:08.557 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:08.557 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:08.558 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:08.558 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:08.976 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:08.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2022-03-01T06:36:08.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2022-03-01T06:36:08.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2022-03-01T06:36:08.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:08.978 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2022-03-01T06:36:09.078 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:09.078 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:09.078 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:09.078 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:09.079 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:09.079 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:09.079 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:09.079 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:09.393 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:09.394 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:09.394 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:09.394 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:09.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:09.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:09.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:09.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:09.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:09.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:09.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:09.812 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:09.813 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:10.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:10.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:10.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2022-03-01T06:36:10.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:10.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2022-03-01T06:36:10.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:10.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:10.450 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:10.450 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:10.451 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:10.451 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:10.451 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:10.452 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:10.764 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:10.765 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:10.765 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:10.766 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:11.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:11.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:11.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:11.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:11.192 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:11.192 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:11.193 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:11.193 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:11.193 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:11.610 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:11.611 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:11.611 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2022-03-01T06:36:11.611 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:11.612 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2022-03-01T06:36:12.012 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:12.013 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:12.013 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:12.013 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:12.014 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:12.014 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:12.015 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:12.015 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:12.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:12.332 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:12.332 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:12.333 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:12.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:12.751 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:12.751 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:12.751 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:12.752 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:12.752 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:12.752 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:12.752 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:12.753 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:13.175 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:13.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:13.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2022-03-01T06:36:13.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:13.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2022-03-01T06:36:13.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:13.978 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:13.978 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:13.978 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:13.978 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:13.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:13.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:13.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:14.305 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:14.305 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:14.306 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:14.306 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:14.724 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:14.724 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:14.724 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:14.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:14.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:14.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:14.725 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:14.725 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:14.726 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:15.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:15.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:15.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2022-03-01T06:36:15.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:15.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2022-03-01T06:36:16.746 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:16.746 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:16.746 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:16.746 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:16.747 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:16.747 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:16.747 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:16.747 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:17.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:17.070 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:17.070 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:17.070 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:17.491 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:17.491 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:17.491 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:17.492 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:17.492 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:17.492 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:17.492 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:17.493 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:17.493 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:17.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:17.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:17.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2022-03-01T06:36:17.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:17.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2022-03-01T06:36:21.136 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:21.137 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:21.137 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:21.137 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:21.137 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:21.138 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:21.138 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:21.138 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:21.459 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:21.459 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:21.459 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:21.459 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:21.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:21.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:21.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:21.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:21.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:21.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:21.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:21.880 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:21.880 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:22.295 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:22.295 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:22.296 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2022-03-01T06:36:22.296 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:22.296 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2022-03-01T06:36:28.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:28.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:28.697 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:28.698 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:28.698 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:28.698 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:28.698 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:28.699 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:29.023 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:29.023 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:29.023 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:29.024 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:29.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:29.449 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:29.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:29.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:29.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:29.450 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:29.451 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:29.451 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:29.451 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:29.888 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:29.889 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:29.889 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2022-03-01T06:36:29.889 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2022-03-01T06:36:29.889 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2022-03-01T06:36:37.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2022-03-01T06:36:37.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:36:37.190 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:36:37.191 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:36:37.191 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:36:37.191 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:36:37.191 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:36:37.192 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:36:37.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2022-03-01T06:36:37.515 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:36:37.515 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:36:37.516 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:36:37.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2022-03-01T06:36:37.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2022-03-01T06:36:37.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2022-03-01T06:36:37.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2022-03-01T06:36:37.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2022-03-01T06:36:37.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2022-03-01T06:36:37.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2022-03-01T06:36:37.937 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2022-03-01T06:36:37.937 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2022-03-01T06:36:38.366 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2022-03-01T06:36:38.367 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2022-03-01T06:36:38.367 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2022-03-01T06:36:38.367 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2022-03-01T06:36:38.805 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:36:38.806 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_fingerprint": "4b101960-99f0-4d69-83c4-35dc2dcfcfdf", 2022-03-01T06:36:38.806 INFO:tasks.workunit.client.0.smithi169.stdout: "version": "15.2.15-249-g45ece157", 2022-03-01T06:36:38.806 INFO:tasks.workunit.client.0.smithi169.stdout: "commit": "45ece157fc29b55a5ee952c81c0c23abdb89bf98", 2022-03-01T06:36:38.806 INFO:tasks.workunit.client.0.smithi169.stdout: "timestamp": "2022-03-01T06:36:38.825834+0000", 2022-03-01T06:36:38.807 INFO:tasks.workunit.client.0.smithi169.stdout: "tag": "", 2022-03-01T06:36:38.807 INFO:tasks.workunit.client.0.smithi169.stdout: "health": { 2022-03-01T06:36:38.807 INFO:tasks.workunit.client.0.smithi169.stdout: "status": "HEALTH_WARN", 2022-03-01T06:36:38.808 INFO:tasks.workunit.client.0.smithi169.stdout: "checks": { 2022-03-01T06:36:38.808 INFO:tasks.workunit.client.0.smithi169.stdout: "OSDMAP_FLAGS": { 2022-03-01T06:36:38.808 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:38.808 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:38.809 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "nodown,noout,noin flag(s) set", 2022-03-01T06:36:38.809 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 17 2022-03-01T06:36:38.809 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.809 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [], 2022-03-01T06:36:38.810 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:38.810 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.810 INFO:tasks.workunit.client.0.smithi169.stdout: "OSD_DOWN": { 2022-03-01T06:36:38.810 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:38.811 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:38.811 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "1 osds down", 2022-03-01T06:36:38.811 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:36:38.811 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.812 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:36:38.812 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.812 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "osd.1 (root=default,host=smithi169) is down" 2022-03-01T06:36:38.812 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.813 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.813 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:38.813 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.813 INFO:tasks.workunit.client.0.smithi169.stdout: "PG_DEGRADED": { 2022-03-01T06:36:38.814 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:38.814 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:38.814 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2022-03-01T06:36:38.814 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 2 2022-03-01T06:36:38.815 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.815 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:36:38.815 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.815 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "pg 1.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [0,2]" 2022-03-01T06:36:38.816 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.816 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.816 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:38.817 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.817 INFO:tasks.workunit.client.0.smithi169.stdout: "POOL_APP_NOT_ENABLED": { 2022-03-01T06:36:38.817 INFO:tasks.workunit.client.0.smithi169.stdout: "severity": "HEALTH_WARN", 2022-03-01T06:36:38.818 INFO:tasks.workunit.client.0.smithi169.stdout: "summary": { 2022-03-01T06:36:38.818 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "1 pool(s) do not have an application enabled", 2022-03-01T06:36:38.819 INFO:tasks.workunit.client.0.smithi169.stdout: "count": 1 2022-03-01T06:36:38.819 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.819 INFO:tasks.workunit.client.0.smithi169.stdout: "detail": [ 2022-03-01T06:36:38.820 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.820 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "application not enabled on pool 'test'" 2022-03-01T06:36:38.820 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.821 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.821 INFO:tasks.workunit.client.0.smithi169.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2022-03-01T06:36:38.822 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.822 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.822 INFO:tasks.workunit.client.0.smithi169.stdout: "muted": false 2022-03-01T06:36:38.823 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.823 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.824 INFO:tasks.workunit.client.0.smithi169.stdout: "mutes": [] 2022-03-01T06:36:38.824 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.825 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap_first_committed": 1, 2022-03-01T06:36:38.825 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap_last_committed": 1, 2022-03-01T06:36:38.825 INFO:tasks.workunit.client.0.smithi169.stdout: "monmap": { 2022-03-01T06:36:38.825 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 1, 2022-03-01T06:36:38.826 INFO:tasks.workunit.client.0.smithi169.stdout: "fsid": "03da9435-ccbd-45e1-be8b-88e9f3ce57a1", 2022-03-01T06:36:38.826 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:34:23.285055Z", 2022-03-01T06:36:38.826 INFO:tasks.workunit.client.0.smithi169.stdout: "created": "2022-03-01T06:34:23.285055Z", 2022-03-01T06:36:38.826 INFO:tasks.workunit.client.0.smithi169.stdout: "min_mon_release": 15, 2022-03-01T06:36:38.827 INFO:tasks.workunit.client.0.smithi169.stdout: "min_mon_release_name": "octopus", 2022-03-01T06:36:38.827 INFO:tasks.workunit.client.0.smithi169.stdout: "features": { 2022-03-01T06:36:38.827 INFO:tasks.workunit.client.0.smithi169.stdout: "persistent": [ 2022-03-01T06:36:38.827 INFO:tasks.workunit.client.0.smithi169.stdout: "kraken", 2022-03-01T06:36:38.828 INFO:tasks.workunit.client.0.smithi169.stdout: "luminous", 2022-03-01T06:36:38.828 INFO:tasks.workunit.client.0.smithi169.stdout: "mimic", 2022-03-01T06:36:38.828 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap-prune", 2022-03-01T06:36:38.829 INFO:tasks.workunit.client.0.smithi169.stdout: "nautilus", 2022-03-01T06:36:38.829 INFO:tasks.workunit.client.0.smithi169.stdout: "octopus" 2022-03-01T06:36:38.829 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.829 INFO:tasks.workunit.client.0.smithi169.stdout: "optional": [] 2022-03-01T06:36:38.830 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.830 INFO:tasks.workunit.client.0.smithi169.stdout: "mons": [ 2022-03-01T06:36:38.830 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.830 INFO:tasks.workunit.client.0.smithi169.stdout: "rank": 0, 2022-03-01T06:36:38.831 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "a", 2022-03-01T06:36:38.831 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:38.832 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.832 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.832 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.833 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:7115", 2022-03-01T06:36:38.833 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 0 2022-03-01T06:36:38.833 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.834 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.834 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.834 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:7115/0", 2022-03-01T06:36:38.835 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:7115/0", 2022-03-01T06:36:38.835 INFO:tasks.workunit.client.0.smithi169.stdout: "priority": 0, 2022-03-01T06:36:38.835 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 0 2022-03-01T06:36:38.835 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.836 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.836 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.836 INFO:tasks.workunit.client.0.smithi169.stdout: "quorum": [ 2022-03-01T06:36:38.836 INFO:tasks.workunit.client.0.smithi169.stdout: 0 2022-03-01T06:36:38.836 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.837 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap": { 2022-03-01T06:36:38.837 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 39, 2022-03-01T06:36:38.837 INFO:tasks.workunit.client.0.smithi169.stdout: "fsid": "03da9435-ccbd-45e1-be8b-88e9f3ce57a1", 2022-03-01T06:36:38.837 INFO:tasks.workunit.client.0.smithi169.stdout: "created": "2022-03-01T06:34:24.009894+0000", 2022-03-01T06:36:38.838 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:35:31.043500+0000", 2022-03-01T06:36:38.838 INFO:tasks.workunit.client.0.smithi169.stdout: "last_up_change": "2022-03-01T06:35:29.579208+0000", 2022-03-01T06:36:38.838 INFO:tasks.workunit.client.0.smithi169.stdout: "last_in_change": "2022-03-01T06:35:30.717028+0000", 2022-03-01T06:36:38.839 INFO:tasks.workunit.client.0.smithi169.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-03-01T06:36:38.839 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_num": 5800384, 2022-03-01T06:36:38.839 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_set": [ 2022-03-01T06:36:38.839 INFO:tasks.workunit.client.0.smithi169.stdout: "nodown", 2022-03-01T06:36:38.840 INFO:tasks.workunit.client.0.smithi169.stdout: "noin", 2022-03-01T06:36:38.840 INFO:tasks.workunit.client.0.smithi169.stdout: "noout", 2022-03-01T06:36:38.840 INFO:tasks.workunit.client.0.smithi169.stdout: "pglog_hardlimit", 2022-03-01T06:36:38.840 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snapdirs", 2022-03-01T06:36:38.841 INFO:tasks.workunit.client.0.smithi169.stdout: "recovery_deletes", 2022-03-01T06:36:38.841 INFO:tasks.workunit.client.0.smithi169.stdout: "sortbitwise" 2022-03-01T06:36:38.841 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.842 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_version": 9, 2022-03-01T06:36:38.842 INFO:tasks.workunit.client.0.smithi169.stdout: "full_ratio": 0.99000000953674316, 2022-03-01T06:36:38.842 INFO:tasks.workunit.client.0.smithi169.stdout: "backfillfull_ratio": 0.99000000953674316, 2022-03-01T06:36:38.842 INFO:tasks.workunit.client.0.smithi169.stdout: "nearfull_ratio": 0.85000002384185791, 2022-03-01T06:36:38.843 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_snapshot": "", 2022-03-01T06:36:38.843 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_max": 1, 2022-03-01T06:36:38.843 INFO:tasks.workunit.client.0.smithi169.stdout: "max_osd": 3, 2022-03-01T06:36:38.843 INFO:tasks.workunit.client.0.smithi169.stdout: "require_min_compat_client": "jewel", 2022-03-01T06:36:38.844 INFO:tasks.workunit.client.0.smithi169.stdout: "min_compat_client": "jewel", 2022-03-01T06:36:38.844 INFO:tasks.workunit.client.0.smithi169.stdout: "require_osd_release": "octopus", 2022-03-01T06:36:38.844 INFO:tasks.workunit.client.0.smithi169.stdout: "pools": [ 2022-03-01T06:36:38.845 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.845 INFO:tasks.workunit.client.0.smithi169.stdout: "pool": 1, 2022-03-01T06:36:38.845 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_name": "test", 2022-03-01T06:36:38.846 INFO:tasks.workunit.client.0.smithi169.stdout: "create_time": "2022-03-01T06:34:44.802792+0000", 2022-03-01T06:36:38.846 INFO:tasks.workunit.client.0.smithi169.stdout: "flags": 1, 2022-03-01T06:36:38.846 INFO:tasks.workunit.client.0.smithi169.stdout: "flags_names": "hashpspool", 2022-03-01T06:36:38.846 INFO:tasks.workunit.client.0.smithi169.stdout: "type": 1, 2022-03-01T06:36:38.847 INFO:tasks.workunit.client.0.smithi169.stdout: "size": 3, 2022-03-01T06:36:38.847 INFO:tasks.workunit.client.0.smithi169.stdout: "min_size": 2, 2022-03-01T06:36:38.847 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_rule": 0, 2022-03-01T06:36:38.847 INFO:tasks.workunit.client.0.smithi169.stdout: "object_hash": 2, 2022-03-01T06:36:38.848 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_autoscale_mode": "off", 2022-03-01T06:36:38.848 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num": 1, 2022-03-01T06:36:38.848 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_placement_num": 1, 2022-03-01T06:36:38.848 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_placement_num_target": 1, 2022-03-01T06:36:38.849 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num_target": 1, 2022-03-01T06:36:38.849 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_num_pending": 1, 2022-03-01T06:36:38.849 INFO:tasks.workunit.client.0.smithi169.stdout: "last_pg_merge_meta": { 2022-03-01T06:36:38.849 INFO:tasks.workunit.client.0.smithi169.stdout: "source_pgid": "0.0", 2022-03-01T06:36:38.850 INFO:tasks.workunit.client.0.smithi169.stdout: "ready_epoch": 0, 2022-03-01T06:36:38.850 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_started": 0, 2022-03-01T06:36:38.851 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_clean": 0, 2022-03-01T06:36:38.851 INFO:tasks.workunit.client.0.smithi169.stdout: "source_version": "0'0", 2022-03-01T06:36:38.851 INFO:tasks.workunit.client.0.smithi169.stdout: "target_version": "0'0" 2022-03-01T06:36:38.851 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.852 INFO:tasks.workunit.client.0.smithi169.stdout: "last_change": "27", 2022-03-01T06:36:38.852 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend": "0", 2022-03-01T06:36:38.852 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2022-03-01T06:36:38.853 INFO:tasks.workunit.client.0.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2022-03-01T06:36:38.853 INFO:tasks.workunit.client.0.smithi169.stdout: "auid": 0, 2022-03-01T06:36:38.853 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_mode": "selfmanaged", 2022-03-01T06:36:38.854 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_seq": 0, 2022-03-01T06:36:38.854 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_epoch": 0, 2022-03-01T06:36:38.855 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_snaps": [], 2022-03-01T06:36:38.855 INFO:tasks.workunit.client.0.smithi169.stdout: "removed_snaps": "[]", 2022-03-01T06:36:38.856 INFO:tasks.workunit.client.0.smithi169.stdout: "quota_max_bytes": 0, 2022-03-01T06:36:38.856 INFO:tasks.workunit.client.0.smithi169.stdout: "quota_max_objects": 0, 2022-03-01T06:36:38.856 INFO:tasks.workunit.client.0.smithi169.stdout: "tiers": [], 2022-03-01T06:36:38.857 INFO:tasks.workunit.client.0.smithi169.stdout: "tier_of": -1, 2022-03-01T06:36:38.857 INFO:tasks.workunit.client.0.smithi169.stdout: "read_tier": -1, 2022-03-01T06:36:38.858 INFO:tasks.workunit.client.0.smithi169.stdout: "write_tier": -1, 2022-03-01T06:36:38.858 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_mode": "none", 2022-03-01T06:36:38.859 INFO:tasks.workunit.client.0.smithi169.stdout: "target_max_bytes": 0, 2022-03-01T06:36:38.859 INFO:tasks.workunit.client.0.smithi169.stdout: "target_max_objects": 0, 2022-03-01T06:36:38.860 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-03-01T06:36:38.860 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-03-01T06:36:38.861 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2022-03-01T06:36:38.861 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_min_flush_age": 0, 2022-03-01T06:36:38.862 INFO:tasks.workunit.client.0.smithi169.stdout: "cache_min_evict_age": 0, 2022-03-01T06:36:38.862 INFO:tasks.workunit.client.0.smithi169.stdout: "erasure_code_profile": "", 2022-03-01T06:36:38.863 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_params": { 2022-03-01T06:36:38.863 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "none" 2022-03-01T06:36:38.864 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.864 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_period": 0, 2022-03-01T06:36:38.865 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_count": 0, 2022-03-01T06:36:38.865 INFO:tasks.workunit.client.0.smithi169.stdout: "use_gmt_hitset": true, 2022-03-01T06:36:38.866 INFO:tasks.workunit.client.0.smithi169.stdout: "min_read_recency_for_promote": 0, 2022-03-01T06:36:38.866 INFO:tasks.workunit.client.0.smithi169.stdout: "min_write_recency_for_promote": 0, 2022-03-01T06:36:38.867 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2022-03-01T06:36:38.867 INFO:tasks.workunit.client.0.smithi169.stdout: "hit_set_search_last_n": 0, 2022-03-01T06:36:38.868 INFO:tasks.workunit.client.0.smithi169.stdout: "grade_table": [], 2022-03-01T06:36:38.868 INFO:tasks.workunit.client.0.smithi169.stdout: "stripe_width": 0, 2022-03-01T06:36:38.868 INFO:tasks.workunit.client.0.smithi169.stdout: "expected_num_objects": 0, 2022-03-01T06:36:38.869 INFO:tasks.workunit.client.0.smithi169.stdout: "fast_read": false, 2022-03-01T06:36:38.869 INFO:tasks.workunit.client.0.smithi169.stdout: "options": {}, 2022-03-01T06:36:38.870 INFO:tasks.workunit.client.0.smithi169.stdout: "application_metadata": {} 2022-03-01T06:36:38.871 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.871 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.872 INFO:tasks.workunit.client.0.smithi169.stdout: "osds": [ 2022-03-01T06:36:38.872 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.873 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:38.873 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8", 2022-03-01T06:36:38.874 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 1, 2022-03-01T06:36:38.874 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:36:38.875 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:36:38.875 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:36:38.876 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 5, 2022-03-01T06:36:38.876 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 27, 2022-03-01T06:36:38.876 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 34, 2022-03-01T06:36:38.877 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 38, 2022-03-01T06:36:38.877 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 28, 2022-03-01T06:36:38.877 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:36:38.877 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:38.877 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.878 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.878 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.878 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6802", 2022-03-01T06:36:38.878 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.879 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.879 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.879 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.880 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6803", 2022-03-01T06:36:38.880 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.880 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.880 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.881 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.881 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:36:38.881 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.881 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.882 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.882 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6804", 2022-03-01T06:36:38.882 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.882 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.883 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.883 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.883 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6805", 2022-03-01T06:36:38.883 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.884 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.884 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.884 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.885 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:36:38.885 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.885 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.885 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.885 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6808", 2022-03-01T06:36:38.886 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.886 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.886 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.887 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.887 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6809", 2022-03-01T06:36:38.887 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.888 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.888 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.888 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.889 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:36:38.889 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.889 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.889 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.890 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6806", 2022-03-01T06:36:38.890 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.890 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.891 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.891 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.891 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6807", 2022-03-01T06:36:38.892 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44358 2022-03-01T06:36:38.892 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.892 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.892 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.893 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6803/44358", 2022-03-01T06:36:38.893 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6805/44358", 2022-03-01T06:36:38.893 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6809/44358", 2022-03-01T06:36:38.894 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6807/44358", 2022-03-01T06:36:38.894 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:36:38.896 INFO:tasks.workunit.client.0.smithi169.stdout: "exists", 2022-03-01T06:36:38.896 INFO:tasks.workunit.client.0.smithi169.stdout: "up" 2022-03-01T06:36:38.897 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.897 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.898 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.898 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:38.898 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "37a8e729-ccba-40e7-90ef-c122436009c4", 2022-03-01T06:36:38.899 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 0, 2022-03-01T06:36:38.899 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:36:38.900 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:36:38.900 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:36:38.900 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 0, 2022-03-01T06:36:38.901 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 0, 2022-03-01T06:36:38.901 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 10, 2022-03-01T06:36:38.902 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 21, 2022-03-01T06:36:38.902 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 29, 2022-03-01T06:36:38.903 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:36:38.903 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:38.903 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.903 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.904 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.904 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6810", 2022-03-01T06:36:38.904 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.904 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.905 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.905 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.905 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6811", 2022-03-01T06:36:38.906 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.906 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.906 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.907 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.907 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:36:38.907 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.907 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.907 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.908 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6812", 2022-03-01T06:36:38.908 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.908 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.908 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.909 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.909 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6813", 2022-03-01T06:36:38.909 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.909 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.910 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.910 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.910 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:36:38.911 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.911 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.911 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.911 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6816", 2022-03-01T06:36:38.912 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.912 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.912 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.912 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.913 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6817", 2022-03-01T06:36:38.913 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.913 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.913 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.914 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.914 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:36:38.914 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.915 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.915 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.916 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6814", 2022-03-01T06:36:38.916 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.916 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.917 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.917 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.918 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6815", 2022-03-01T06:36:38.918 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 39958 2022-03-01T06:36:38.918 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.919 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.919 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.919 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6811/39958", 2022-03-01T06:36:38.919 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6813/39958", 2022-03-01T06:36:38.920 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39958", 2022-03-01T06:36:38.920 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39958", 2022-03-01T06:36:38.920 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:36:38.920 INFO:tasks.workunit.client.0.smithi169.stdout: "exists" 2022-03-01T06:36:38.921 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.921 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.922 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.922 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:38.922 INFO:tasks.workunit.client.0.smithi169.stdout: "uuid": "dfb1e4b2-84de-4539-b25a-a5ab79b41e6d", 2022-03-01T06:36:38.923 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 1, 2022-03-01T06:36:38.923 INFO:tasks.workunit.client.0.smithi169.stdout: "in": 1, 2022-03-01T06:36:38.923 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 1, 2022-03-01T06:36:38.923 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_affinity": 1, 2022-03-01T06:36:38.924 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_begin": 14, 2022-03-01T06:36:38.924 INFO:tasks.workunit.client.0.smithi169.stdout: "last_clean_end": 27, 2022-03-01T06:36:38.924 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 36, 2022-03-01T06:36:38.924 INFO:tasks.workunit.client.0.smithi169.stdout: "up_thru": 0, 2022-03-01T06:36:38.925 INFO:tasks.workunit.client.0.smithi169.stdout: "down_at": 30, 2022-03-01T06:36:38.925 INFO:tasks.workunit.client.0.smithi169.stdout: "lost_at": 0, 2022-03-01T06:36:38.925 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addrs": { 2022-03-01T06:36:38.925 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.925 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.926 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.926 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6810", 2022-03-01T06:36:38.926 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.926 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.927 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.927 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.927 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6811", 2022-03-01T06:36:38.927 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.928 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.928 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.929 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.929 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addrs": { 2022-03-01T06:36:38.929 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.930 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.930 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.931 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6812", 2022-03-01T06:36:38.931 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.932 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.932 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.933 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.933 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6813", 2022-03-01T06:36:38.933 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.934 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.934 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.934 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.935 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addrs": { 2022-03-01T06:36:38.935 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.936 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.936 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.936 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6816", 2022-03-01T06:36:38.936 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.937 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.937 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.938 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.938 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6817", 2022-03-01T06:36:38.938 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.939 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.939 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.940 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.940 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addrs": { 2022-03-01T06:36:38.940 INFO:tasks.workunit.client.0.smithi169.stdout: "addrvec": [ 2022-03-01T06:36:38.941 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.941 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v2", 2022-03-01T06:36:38.941 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6814", 2022-03-01T06:36:38.941 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.941 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.942 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.942 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "v1", 2022-03-01T06:36:38.942 INFO:tasks.workunit.client.0.smithi169.stdout: "addr": "127.0.0.1:6815", 2022-03-01T06:36:38.942 INFO:tasks.workunit.client.0.smithi169.stdout: "nonce": 44552 2022-03-01T06:36:38.943 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.943 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.943 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.943 INFO:tasks.workunit.client.0.smithi169.stdout: "public_addr": "127.0.0.1:6811/44552", 2022-03-01T06:36:38.944 INFO:tasks.workunit.client.0.smithi169.stdout: "cluster_addr": "127.0.0.1:6813/44552", 2022-03-01T06:36:38.944 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_back_addr": "127.0.0.1:6817/44552", 2022-03-01T06:36:38.944 INFO:tasks.workunit.client.0.smithi169.stdout: "heartbeat_front_addr": "127.0.0.1:6815/44552", 2022-03-01T06:36:38.944 INFO:tasks.workunit.client.0.smithi169.stdout: "state": [ 2022-03-01T06:36:38.944 INFO:tasks.workunit.client.0.smithi169.stdout: "exists", 2022-03-01T06:36:38.945 INFO:tasks.workunit.client.0.smithi169.stdout: "up" 2022-03-01T06:36:38.945 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:38.945 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.945 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.946 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_xinfo": [ 2022-03-01T06:36:38.946 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.946 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:38.946 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:35:23.287456+0000", 2022-03-01T06:36:38.947 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:36:38.947 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:36:38.947 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:36:38.948 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:36:38.948 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:34:30.918532+0000", 2022-03-01T06:36:38.948 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:36:38.948 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.949 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.949 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:38.949 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:35:23.759955+0000", 2022-03-01T06:36:38.950 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:36:38.950 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:36:38.950 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:36:38.950 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:36:38.950 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:34:36.971000+0000", 2022-03-01T06:36:38.951 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:36:38.951 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.951 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.951 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:38.952 INFO:tasks.workunit.client.0.smithi169.stdout: "down_stamp": "2022-03-01T06:35:24.290824+0000", 2022-03-01T06:36:38.952 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_probability": 0, 2022-03-01T06:36:38.952 INFO:tasks.workunit.client.0.smithi169.stdout: "laggy_interval": 0, 2022-03-01T06:36:38.952 INFO:tasks.workunit.client.0.smithi169.stdout: "features": 4540138292840890367, 2022-03-01T06:36:38.952 INFO:tasks.workunit.client.0.smithi169.stdout: "old_weight": 0, 2022-03-01T06:36:38.953 INFO:tasks.workunit.client.0.smithi169.stdout: "last_purged_snaps_scrub": "2022-03-01T06:34:42.935271+0000", 2022-03-01T06:36:38.953 INFO:tasks.workunit.client.0.smithi169.stdout: "dead_epoch": 0 2022-03-01T06:36:38.953 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.953 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.954 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_upmap": [], 2022-03-01T06:36:38.954 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_upmap_items": [], 2022-03-01T06:36:38.954 INFO:tasks.workunit.client.0.smithi169.stdout: "pg_temp": [], 2022-03-01T06:36:38.954 INFO:tasks.workunit.client.0.smithi169.stdout: "primary_temp": [], 2022-03-01T06:36:38.954 INFO:tasks.workunit.client.0.smithi169.stdout: "blacklist": {}, 2022-03-01T06:36:38.955 INFO:tasks.workunit.client.0.smithi169.stdout: "erasure_code_profiles": { 2022-03-01T06:36:38.955 INFO:tasks.workunit.client.0.smithi169.stdout: "default": { 2022-03-01T06:36:38.955 INFO:tasks.workunit.client.0.smithi169.stdout: "k": "2", 2022-03-01T06:36:38.955 INFO:tasks.workunit.client.0.smithi169.stdout: "m": "2", 2022-03-01T06:36:38.956 INFO:tasks.workunit.client.0.smithi169.stdout: "plugin": "jerasure", 2022-03-01T06:36:38.956 INFO:tasks.workunit.client.0.smithi169.stdout: "technique": "reed_sol_van" 2022-03-01T06:36:38.956 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.956 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.956 INFO:tasks.workunit.client.0.smithi169.stdout: "removed_snaps_queue": [], 2022-03-01T06:36:38.957 INFO:tasks.workunit.client.0.smithi169.stdout: "new_removed_snaps": [], 2022-03-01T06:36:38.957 INFO:tasks.workunit.client.0.smithi169.stdout: "new_purged_snaps": [], 2022-03-01T06:36:38.957 INFO:tasks.workunit.client.0.smithi169.stdout: "crush_node_flags": {}, 2022-03-01T06:36:38.957 INFO:tasks.workunit.client.0.smithi169.stdout: "device_class_flags": {} 2022-03-01T06:36:38.958 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.958 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_metadata": [ 2022-03-01T06:36:38.958 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.958 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:38.959 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:36:38.959 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6804/44358,v1:127.0.0.1:6805/44358]", 2022-03-01T06:36:38.959 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:36:38.959 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:36:38.959 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:36:38.960 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:36:38.960 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:36:38.960 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:36:38.960 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:36:38.961 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:36:38.961 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:36:38.961 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2022-03-01T06:36:38.961 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:36:38.962 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:36:38.962 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:36:38.962 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:36:38.962 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:36:38.963 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:36:38.963 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:36:38.963 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:36:38.963 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:36:38.963 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:36:38.964 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:36:38.964 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:36:38.964 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:36:38.964 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:36:38.965 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:36:38.965 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6802/44358,v1:127.0.0.1:6803/44358]", 2022-03-01T06:36:38.965 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:36:38.965 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/44358,v1:127.0.0.1:6809/44358]", 2022-03-01T06:36:38.965 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/44358,v1:127.0.0.1:6807/44358]", 2022-03-01T06:36:38.966 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:36:38.966 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:36:38.966 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:36:38.966 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:36:38.967 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:36:38.967 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:36:38.967 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:36:38.967 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:36:38.968 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:36:38.968 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/0", 2022-03-01T06:36:38.968 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:36:38.968 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:36:38.968 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:36:38.969 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.969 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.969 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:38.969 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:36:38.969 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6812/39958,v1:127.0.0.1:6813/39958]", 2022-03-01T06:36:38.970 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:36:38.970 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:36:38.970 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:36:38.970 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:36:38.971 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:36:38.971 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:36:38.971 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:36:38.971 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:36:38.972 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:36:38.972 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2022-03-01T06:36:38.972 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:36:38.972 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:36:38.972 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:36:38.973 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:36:38.973 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:36:38.973 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:36:38.973 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:36:38.974 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:36:38.974 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:36:38.974 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:36:38.974 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:36:38.975 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:36:38.975 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:36:38.975 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:36:38.975 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:36:38.975 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6810/39958,v1:127.0.0.1:6811/39958]", 2022-03-01T06:36:38.976 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:36:38.976 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39958,v1:127.0.0.1:6817/39958]", 2022-03-01T06:36:38.976 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39958,v1:127.0.0.1:6815/39958]", 2022-03-01T06:36:38.976 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:36:38.976 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:36:38.977 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:36:38.977 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:36:38.977 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:36:38.977 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:36:38.978 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:36:38.978 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:36:38.978 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:36:38.978 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/1", 2022-03-01T06:36:38.979 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:36:38.979 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:36:38.980 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:36:38.980 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.980 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.981 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:38.981 INFO:tasks.workunit.client.0.smithi169.stdout: "arch": "x86_64", 2022-03-01T06:36:38.982 INFO:tasks.workunit.client.0.smithi169.stdout: "back_addr": "[v2:127.0.0.1:6812/44552,v1:127.0.0.1:6813/44552]", 2022-03-01T06:36:38.982 INFO:tasks.workunit.client.0.smithi169.stdout: "back_iface": "", 2022-03-01T06:36:38.983 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs": "1", 2022-03-01T06:36:38.983 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_db": "0", 2022-03-01T06:36:38.983 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_dedicated_wal": "0", 2022-03-01T06:36:38.984 INFO:tasks.workunit.client.0.smithi169.stdout: "bluefs_single_shared_device": "1", 2022-03-01T06:36:38.984 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_access_mode": "file", 2022-03-01T06:36:38.984 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_block_size": "4096", 2022-03-01T06:36:38.984 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_devices": "sda", 2022-03-01T06:36:38.985 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_driver": "KernelDevice", 2022-03-01T06:36:38.985 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2022-03-01T06:36:38.986 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_rotational": "1", 2022-03-01T06:36:38.986 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_size": "107374182400", 2022-03-01T06:36:38.986 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_support_discard": "0", 2022-03-01T06:36:38.987 INFO:tasks.workunit.client.0.smithi169.stdout: "bluestore_bdev_type": "hdd", 2022-03-01T06:36:38.987 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_release": "octopus", 2022-03-01T06:36:38.987 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version": "ceph version 15.2.15-249-g45ece157 (45ece157fc29b55a5ee952c81c0c23abdb89bf98) octopus (stable)", 2022-03-01T06:36:38.988 INFO:tasks.workunit.client.0.smithi169.stdout: "ceph_version_short": "15.2.15-249-g45ece157", 2022-03-01T06:36:38.988 INFO:tasks.workunit.client.0.smithi169.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2022-03-01T06:36:38.988 INFO:tasks.workunit.client.0.smithi169.stdout: "default_device_class": "hdd", 2022-03-01T06:36:38.988 INFO:tasks.workunit.client.0.smithi169.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XFL1", 2022-03-01T06:36:38.988 INFO:tasks.workunit.client.0.smithi169.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2022-03-01T06:36:38.989 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": "sda", 2022-03-01T06:36:38.989 INFO:tasks.workunit.client.0.smithi169.stdout: "distro": "centos", 2022-03-01T06:36:38.989 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_description": "CentOS Stream 8", 2022-03-01T06:36:38.989 INFO:tasks.workunit.client.0.smithi169.stdout: "distro_version": "8", 2022-03-01T06:36:38.990 INFO:tasks.workunit.client.0.smithi169.stdout: "front_addr": "[v2:127.0.0.1:6810/44552,v1:127.0.0.1:6811/44552]", 2022-03-01T06:36:38.990 INFO:tasks.workunit.client.0.smithi169.stdout: "front_iface": "", 2022-03-01T06:36:38.990 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/44552,v1:127.0.0.1:6817/44552]", 2022-03-01T06:36:38.990 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/44552,v1:127.0.0.1:6815/44552]", 2022-03-01T06:36:38.991 INFO:tasks.workunit.client.0.smithi169.stdout: "hostname": "smithi169", 2022-03-01T06:36:38.991 INFO:tasks.workunit.client.0.smithi169.stdout: "journal_rotational": "1", 2022-03-01T06:36:38.991 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022", 2022-03-01T06:36:38.991 INFO:tasks.workunit.client.0.smithi169.stdout: "kernel_version": "4.18.0-365.el8.x86_64", 2022-03-01T06:36:38.992 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_swap_kb": "0", 2022-03-01T06:36:38.992 INFO:tasks.workunit.client.0.smithi169.stdout: "mem_total_kb": "32605572", 2022-03-01T06:36:38.992 INFO:tasks.workunit.client.0.smithi169.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2022-03-01T06:36:38.993 INFO:tasks.workunit.client.0.smithi169.stdout: "objectstore_numa_unknown_devices": "sda", 2022-03-01T06:36:38.993 INFO:tasks.workunit.client.0.smithi169.stdout: "os": "Linux", 2022-03-01T06:36:38.994 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_data": "td/divergent-priors/2", 2022-03-01T06:36:38.994 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_objectstore": "bluestore", 2022-03-01T06:36:38.994 INFO:tasks.workunit.client.0.smithi169.stdout: "osdspec_affinity": "", 2022-03-01T06:36:38.995 INFO:tasks.workunit.client.0.smithi169.stdout: "rotational": "1" 2022-03-01T06:36:38.995 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:38.996 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:38.996 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_clean_epochs": { 2022-03-01T06:36:38.996 INFO:tasks.workunit.client.0.smithi169.stdout: "min_last_epoch_clean": 0, 2022-03-01T06:36:38.997 INFO:tasks.workunit.client.0.smithi169.stdout: "last_epoch_clean": { 2022-03-01T06:36:38.997 INFO:tasks.workunit.client.0.smithi169.stdout: "per_pool": [] 2022-03-01T06:36:38.997 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.997 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_epochs": [ 2022-03-01T06:36:38.998 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.998 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:38.998 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 34 2022-03-01T06:36:38.998 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:38.999 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:38.999 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:38.999 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 10 2022-03-01T06:36:38.999 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.000 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.000 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:39.000 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 36 2022-03-01T06:36:39.001 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.001 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:39.001 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.002 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_first_committed": 1, 2022-03-01T06:36:39.002 INFO:tasks.workunit.client.0.smithi169.stdout: "osdmap_last_committed": 39, 2022-03-01T06:36:39.002 INFO:tasks.workunit.client.0.smithi169.stdout: "crushmap": { 2022-03-01T06:36:39.002 INFO:tasks.workunit.client.0.smithi169.stdout: "devices": [ 2022-03-01T06:36:39.003 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.003 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:39.004 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.0", 2022-03-01T06:36:39.004 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:36:39.004 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.004 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.005 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:39.005 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.1", 2022-03-01T06:36:39.005 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:36:39.006 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.006 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.006 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:39.006 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd.2", 2022-03-01T06:36:39.007 INFO:tasks.workunit.client.0.smithi169.stdout: "class": "hdd" 2022-03-01T06:36:39.007 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.007 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.007 INFO:tasks.workunit.client.0.smithi169.stdout: "types": [ 2022-03-01T06:36:39.008 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.008 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 0, 2022-03-01T06:36:39.009 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "osd" 2022-03-01T06:36:39.009 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.009 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.009 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:36:39.010 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "host" 2022-03-01T06:36:39.010 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.010 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.010 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 2, 2022-03-01T06:36:39.011 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "chassis" 2022-03-01T06:36:39.011 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.011 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.011 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 3, 2022-03-01T06:36:39.012 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "rack" 2022-03-01T06:36:39.012 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.012 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.012 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 4, 2022-03-01T06:36:39.012 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "row" 2022-03-01T06:36:39.013 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.013 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.013 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 5, 2022-03-01T06:36:39.014 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "pdu" 2022-03-01T06:36:39.014 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.015 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.015 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 6, 2022-03-01T06:36:39.015 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "pod" 2022-03-01T06:36:39.016 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.016 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.016 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 7, 2022-03-01T06:36:39.016 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "room" 2022-03-01T06:36:39.017 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.017 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.017 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 8, 2022-03-01T06:36:39.017 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "datacenter" 2022-03-01T06:36:39.017 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.018 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.018 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 9, 2022-03-01T06:36:39.018 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "zone" 2022-03-01T06:36:39.018 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.019 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.019 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 10, 2022-03-01T06:36:39.019 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "region" 2022-03-01T06:36:39.020 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.020 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.024 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:36:39.024 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "root" 2022-03-01T06:36:39.025 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.025 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.025 INFO:tasks.workunit.client.0.smithi169.stdout: "buckets": [ 2022-03-01T06:36:39.025 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.026 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -1, 2022-03-01T06:36:39.026 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "default", 2022-03-01T06:36:39.026 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:36:39.026 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "root", 2022-03-01T06:36:39.027 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:39.027 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:39.028 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:39.028 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:39.028 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.029 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -3, 2022-03-01T06:36:39.029 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:39.029 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:39.030 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.030 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:39.030 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.030 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.031 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -2, 2022-03-01T06:36:39.031 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "default~hdd", 2022-03-01T06:36:39.031 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 11, 2022-03-01T06:36:39.032 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "root", 2022-03-01T06:36:39.032 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:39.032 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:39.033 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:39.033 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:39.033 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.034 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -4, 2022-03-01T06:36:39.034 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:39.035 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:39.035 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.035 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:39.036 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.036 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.037 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -3, 2022-03-01T06:36:39.037 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "smithi169", 2022-03-01T06:36:39.037 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:36:39.038 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "host", 2022-03-01T06:36:39.038 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:39.038 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:39.038 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:39.039 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:39.039 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.039 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:39.039 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:39.040 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:39.040 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.040 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.040 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:39.040 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:39.041 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 1 2022-03-01T06:36:39.041 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.041 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.042 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:39.042 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:39.042 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 2 2022-03-01T06:36:39.042 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.043 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:39.043 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.043 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.043 INFO:tasks.workunit.client.0.smithi169.stdout: "id": -4, 2022-03-01T06:36:39.043 INFO:tasks.workunit.client.0.smithi169.stdout: "name": "smithi169~hdd", 2022-03-01T06:36:39.044 INFO:tasks.workunit.client.0.smithi169.stdout: "type_id": 1, 2022-03-01T06:36:39.044 INFO:tasks.workunit.client.0.smithi169.stdout: "type_name": "host", 2022-03-01T06:36:39.044 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 19206, 2022-03-01T06:36:39.044 INFO:tasks.workunit.client.0.smithi169.stdout: "alg": "straw2", 2022-03-01T06:36:39.045 INFO:tasks.workunit.client.0.smithi169.stdout: "hash": "rjenkins1", 2022-03-01T06:36:39.045 INFO:tasks.workunit.client.0.smithi169.stdout: "items": [ 2022-03-01T06:36:39.045 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.045 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 0, 2022-03-01T06:36:39.046 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:39.046 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 0 2022-03-01T06:36:39.046 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.046 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.046 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 1, 2022-03-01T06:36:39.047 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:39.047 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 1 2022-03-01T06:36:39.047 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.048 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.048 INFO:tasks.workunit.client.0.smithi169.stdout: "id": 2, 2022-03-01T06:36:39.048 INFO:tasks.workunit.client.0.smithi169.stdout: "weight": 6402, 2022-03-01T06:36:39.048 INFO:tasks.workunit.client.0.smithi169.stdout: "pos": 2 2022-03-01T06:36:39.048 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.049 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:39.049 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.049 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.049 INFO:tasks.workunit.client.0.smithi169.stdout: "rules": [ 2022-03-01T06:36:39.050 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.050 INFO:tasks.workunit.client.0.smithi169.stdout: "rule_id": 0, 2022-03-01T06:36:39.050 INFO:tasks.workunit.client.0.smithi169.stdout: "rule_name": "replicated_rule", 2022-03-01T06:36:39.051 INFO:tasks.workunit.client.0.smithi169.stdout: "ruleset": 0, 2022-03-01T06:36:39.051 INFO:tasks.workunit.client.0.smithi169.stdout: "type": 1, 2022-03-01T06:36:39.051 INFO:tasks.workunit.client.0.smithi169.stdout: "min_size": 1, 2022-03-01T06:36:39.051 INFO:tasks.workunit.client.0.smithi169.stdout: "max_size": 10, 2022-03-01T06:36:39.051 INFO:tasks.workunit.client.0.smithi169.stdout: "steps": [ 2022-03-01T06:36:39.052 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.052 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "take", 2022-03-01T06:36:39.052 INFO:tasks.workunit.client.0.smithi169.stdout: "item": -1, 2022-03-01T06:36:39.052 INFO:tasks.workunit.client.0.smithi169.stdout: "item_name": "default" 2022-03-01T06:36:39.053 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.053 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.053 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "choose_firstn", 2022-03-01T06:36:39.053 INFO:tasks.workunit.client.0.smithi169.stdout: "num": 0, 2022-03-01T06:36:39.054 INFO:tasks.workunit.client.0.smithi169.stdout: "type": "osd" 2022-03-01T06:36:39.054 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.054 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.055 INFO:tasks.workunit.client.0.smithi169.stdout: "op": "emit" 2022-03-01T06:36:39.055 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.055 INFO:tasks.workunit.client.0.smithi169.stdout: ] 2022-03-01T06:36:39.055 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.055 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.056 INFO:tasks.workunit.client.0.smithi169.stdout: "tunables": { 2022-03-01T06:36:39.056 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_local_tries": 0, 2022-03-01T06:36:39.056 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_local_fallback_tries": 0, 2022-03-01T06:36:39.056 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_total_tries": 50, 2022-03-01T06:36:39.057 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_descend_once": 1, 2022-03-01T06:36:39.057 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_vary_r": 1, 2022-03-01T06:36:39.057 INFO:tasks.workunit.client.0.smithi169.stdout: "chooseleaf_stable": 1, 2022-03-01T06:36:39.057 INFO:tasks.workunit.client.0.smithi169.stdout: "straw_calc_version": 1, 2022-03-01T06:36:39.057 INFO:tasks.workunit.client.0.smithi169.stdout: "allowed_bucket_algs": 54, 2022-03-01T06:36:39.058 INFO:tasks.workunit.client.0.smithi169.stdout: "profile": "jewel", 2022-03-01T06:36:39.058 INFO:tasks.workunit.client.0.smithi169.stdout: "optimal_tunables": 1, 2022-03-01T06:36:39.058 INFO:tasks.workunit.client.0.smithi169.stdout: "legacy_tunables": 0, 2022-03-01T06:36:39.058 INFO:tasks.workunit.client.0.smithi169.stdout: "minimum_required_version": "jewel", 2022-03-01T06:36:39.059 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables": 1, 2022-03-01T06:36:39.059 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables2": 1, 2022-03-01T06:36:39.059 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v2_rules": 0, 2022-03-01T06:36:39.059 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables3": 1, 2022-03-01T06:36:39.060 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v3_rules": 0, 2022-03-01T06:36:39.060 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v4_buckets": 1, 2022-03-01T06:36:39.060 INFO:tasks.workunit.client.0.smithi169.stdout: "require_feature_tunables5": 1, 2022-03-01T06:36:39.060 INFO:tasks.workunit.client.0.smithi169.stdout: "has_v5_rules": 0 2022-03-01T06:36:39.061 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.061 INFO:tasks.workunit.client.0.smithi169.stdout: "choose_args": {} 2022-03-01T06:36:39.061 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.061 INFO:tasks.workunit.client.0.smithi169.stdout: "fsmap": { 2022-03-01T06:36:39.061 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 1, 2022-03-01T06:36:39.062 INFO:tasks.workunit.client.0.smithi169.stdout: "default_fscid": -1, 2022-03-01T06:36:39.062 INFO:tasks.workunit.client.0.smithi169.stdout: "compat": { 2022-03-01T06:36:39.062 INFO:tasks.workunit.client.0.smithi169.stdout: "compat": {}, 2022-03-01T06:36:39.063 INFO:tasks.workunit.client.0.smithi169.stdout: "ro_compat": {}, 2022-03-01T06:36:39.063 INFO:tasks.workunit.client.0.smithi169.stdout: "incompat": { 2022-03-01T06:36:39.063 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_1": "base v0.20", 2022-03-01T06:36:39.063 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_2": "client writeable ranges", 2022-03-01T06:36:39.064 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_3": "default file layouts on dirs", 2022-03-01T06:36:39.064 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_4": "dir inode in separate object", 2022-03-01T06:36:39.064 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_5": "mds uses versioned encoding", 2022-03-01T06:36:39.064 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_6": "dirfrag is stored in omap", 2022-03-01T06:36:39.065 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_8": "no anchor table", 2022-03-01T06:36:39.065 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_9": "file layout v2", 2022-03-01T06:36:39.065 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_10": "snaprealm v2" 2022-03-01T06:36:39.065 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.065 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.066 INFO:tasks.workunit.client.0.smithi169.stdout: "feature_flags": { 2022-03-01T06:36:39.066 INFO:tasks.workunit.client.0.smithi169.stdout: "enable_multiple": false, 2022-03-01T06:36:39.066 INFO:tasks.workunit.client.0.smithi169.stdout: "ever_enabled_multiple": false 2022-03-01T06:36:39.066 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.067 INFO:tasks.workunit.client.0.smithi169.stdout: "standbys": [], 2022-03-01T06:36:39.067 INFO:tasks.workunit.client.0.smithi169.stdout: "filesystems": [] 2022-03-01T06:36:39.067 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.067 INFO:tasks.workunit.client.0.smithi169.stdout: "mdsmap_first_committed": 1, 2022-03-01T06:36:39.067 INFO:tasks.workunit.client.0.smithi169.stdout: "mdsmap_last_committed": 1, 2022-03-01T06:36:39.068 INFO:tasks.workunit.client.0.smithi169.stdout: "auth": { 2022-03-01T06:36:39.068 INFO:tasks.workunit.client.0.smithi169.stdout: "first_committed": 1, 2022-03-01T06:36:39.068 INFO:tasks.workunit.client.0.smithi169.stdout: "last_committed": 5, 2022-03-01T06:36:39.068 INFO:tasks.workunit.client.0.smithi169.stdout: "num_secrets": 3 2022-03-01T06:36:39.069 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.069 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg": 1, 2022-03-01T06:36:39.069 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_active": 1, 2022-03-01T06:36:39.069 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_unknown": 0, 2022-03-01T06:36:39.069 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osd": 3, 2022-03-01T06:36:39.070 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_sum": { 2022-03-01T06:36:39.070 INFO:tasks.workunit.client.0.smithi169.stdout: "stat_sum": { 2022-03-01T06:36:39.070 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes": 47947, 2022-03-01T06:36:39.070 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects": 100, 2022-03-01T06:36:39.071 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_clones": 0, 2022-03-01T06:36:39.071 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_copies": 300, 2022-03-01T06:36:39.071 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing_on_primary": 0, 2022-03-01T06:36:39.071 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing": 0, 2022-03-01T06:36:39.071 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_degraded": 100, 2022-03-01T06:36:39.072 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_misplaced": 0, 2022-03-01T06:36:39.072 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_unfound": 0, 2022-03-01T06:36:39.072 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_dirty": 100, 2022-03-01T06:36:39.072 INFO:tasks.workunit.client.0.smithi169.stdout: "num_whiteouts": 0, 2022-03-01T06:36:39.073 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read": 0, 2022-03-01T06:36:39.073 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read_kb": 0, 2022-03-01T06:36:39.073 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write": 101, 2022-03-01T06:36:39.073 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write_kb": 101, 2022-03-01T06:36:39.073 INFO:tasks.workunit.client.0.smithi169.stdout: "num_scrub_errors": 0, 2022-03-01T06:36:39.074 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shallow_scrub_errors": 0, 2022-03-01T06:36:39.074 INFO:tasks.workunit.client.0.smithi169.stdout: "num_deep_scrub_errors": 0, 2022-03-01T06:36:39.074 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_recovered": 0, 2022-03-01T06:36:39.074 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_recovered": 0, 2022-03-01T06:36:39.075 INFO:tasks.workunit.client.0.smithi169.stdout: "num_keys_recovered": 0, 2022-03-01T06:36:39.075 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_omap": 0, 2022-03-01T06:36:39.075 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_hit_set_archive": 0, 2022-03-01T06:36:39.075 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_hit_set_archive": 0, 2022-03-01T06:36:39.076 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush": 0, 2022-03-01T06:36:39.076 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_kb": 0, 2022-03-01T06:36:39.077 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict": 0, 2022-03-01T06:36:39.077 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_kb": 0, 2022-03-01T06:36:39.077 INFO:tasks.workunit.client.0.smithi169.stdout: "num_promote": 0, 2022-03-01T06:36:39.077 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_high": 0, 2022-03-01T06:36:39.078 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_low": 0, 2022-03-01T06:36:39.078 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_some": 0, 2022-03-01T06:36:39.078 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_full": 0, 2022-03-01T06:36:39.078 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_pinned": 0, 2022-03-01T06:36:39.079 INFO:tasks.workunit.client.0.smithi169.stdout: "num_legacy_snapsets": 0, 2022-03-01T06:36:39.079 INFO:tasks.workunit.client.0.smithi169.stdout: "num_large_omap_objects": 0, 2022-03-01T06:36:39.079 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_manifest": 0, 2022-03-01T06:36:39.079 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_bytes": 0, 2022-03-01T06:36:39.080 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_keys": 0, 2022-03-01T06:36:39.080 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_repaired": 0 2022-03-01T06:36:39.080 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.080 INFO:tasks.workunit.client.0.smithi169.stdout: "store_stats": { 2022-03-01T06:36:39.081 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 0, 2022-03-01T06:36:39.081 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 0, 2022-03-01T06:36:39.081 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 0, 2022-03-01T06:36:39.081 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 0, 2022-03-01T06:36:39.082 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 0, 2022-03-01T06:36:39.082 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:39.082 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:39.082 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:39.083 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 0, 2022-03-01T06:36:39.083 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 0 2022-03-01T06:36:39.083 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.083 INFO:tasks.workunit.client.0.smithi169.stdout: "log_size": 17, 2022-03-01T06:36:39.084 INFO:tasks.workunit.client.0.smithi169.stdout: "ondisk_log_size": 17, 2022-03-01T06:36:39.084 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 2, 2022-03-01T06:36:39.084 INFO:tasks.workunit.client.0.smithi169.stdout: "acting": 2, 2022-03-01T06:36:39.084 INFO:tasks.workunit.client.0.smithi169.stdout: "num_store_stats": 0 2022-03-01T06:36:39.084 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.085 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_sum": { 2022-03-01T06:36:39.085 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 0, 2022-03-01T06:36:39.085 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 0, 2022-03-01T06:36:39.085 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pgs": 3, 2022-03-01T06:36:39.086 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osds": 3, 2022-03-01T06:36:39.086 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_osds": 3, 2022-03-01T06:36:39.086 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_omap_osds": 3, 2022-03-01T06:36:39.086 INFO:tasks.workunit.client.0.smithi169.stdout: "kb": 314572800, 2022-03-01T06:36:39.086 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used": 3178944, 2022-03-01T06:36:39.087 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_data": 32896, 2022-03-01T06:36:39.087 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_omap": 12, 2022-03-01T06:36:39.087 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_meta": 3145715, 2022-03-01T06:36:39.087 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_avail": 311393856, 2022-03-01T06:36:39.088 INFO:tasks.workunit.client.0.smithi169.stdout: "statfs": { 2022-03-01T06:36:39.088 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 322122547200, 2022-03-01T06:36:39.088 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 318867308544, 2022-03-01T06:36:39.088 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 3221225472, 2022-03-01T06:36:39.088 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 33685504, 2022-03-01T06:36:39.089 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 403046, 2022-03-01T06:36:39.089 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:39.089 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:39.089 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:39.090 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 13214, 2022-03-01T06:36:39.090 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 3221212258 2022-03-01T06:36:39.090 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.090 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_peers": [], 2022-03-01T06:36:39.091 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_trim_queue_len": 0, 2022-03-01T06:36:39.091 INFO:tasks.workunit.client.0.smithi169.stdout: "num_snap_trimming": 0, 2022-03-01T06:36:39.091 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shards_repaired": 0, 2022-03-01T06:36:39.091 INFO:tasks.workunit.client.0.smithi169.stdout: "op_queue_age_hist": { 2022-03-01T06:36:39.091 INFO:tasks.workunit.client.0.smithi169.stdout: "histogram": [], 2022-03-01T06:36:39.092 INFO:tasks.workunit.client.0.smithi169.stdout: "upper_bound": 1 2022-03-01T06:36:39.092 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.092 INFO:tasks.workunit.client.0.smithi169.stdout: "perf_stat": { 2022-03-01T06:36:39.092 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ms": 0, 2022-03-01T06:36:39.092 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ms": 0, 2022-03-01T06:36:39.093 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ns": 0, 2022-03-01T06:36:39.093 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ns": 0 2022-03-01T06:36:39.093 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.093 INFO:tasks.workunit.client.0.smithi169.stdout: "alerts": [], 2022-03-01T06:36:39.094 INFO:tasks.workunit.client.0.smithi169.stdout: "network_ping_times": [] 2022-03-01T06:36:39.094 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.094 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_sum_by_class": { 2022-03-01T06:36:39.094 INFO:tasks.workunit.client.0.smithi169.stdout: "hdd": { 2022-03-01T06:36:39.095 INFO:tasks.workunit.client.0.smithi169.stdout: "up_from": 0, 2022-03-01T06:36:39.095 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 0, 2022-03-01T06:36:39.095 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pgs": 3, 2022-03-01T06:36:39.095 INFO:tasks.workunit.client.0.smithi169.stdout: "num_osds": 3, 2022-03-01T06:36:39.095 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_osds": 3, 2022-03-01T06:36:39.096 INFO:tasks.workunit.client.0.smithi169.stdout: "num_per_pool_omap_osds": 3, 2022-03-01T06:36:39.096 INFO:tasks.workunit.client.0.smithi169.stdout: "kb": 314572800, 2022-03-01T06:36:39.096 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used": 3178944, 2022-03-01T06:36:39.096 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_data": 32896, 2022-03-01T06:36:39.097 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_omap": 12, 2022-03-01T06:36:39.097 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_used_meta": 3145715, 2022-03-01T06:36:39.097 INFO:tasks.workunit.client.0.smithi169.stdout: "kb_avail": 311393856, 2022-03-01T06:36:39.097 INFO:tasks.workunit.client.0.smithi169.stdout: "statfs": { 2022-03-01T06:36:39.098 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 322122547200, 2022-03-01T06:36:39.098 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 318867308544, 2022-03-01T06:36:39.098 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 3221225472, 2022-03-01T06:36:39.098 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 33685504, 2022-03-01T06:36:39.098 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 403046, 2022-03-01T06:36:39.099 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:39.099 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:39.099 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:39.099 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 13214, 2022-03-01T06:36:39.100 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 3221212258 2022-03-01T06:36:39.100 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.100 INFO:tasks.workunit.client.0.smithi169.stdout: "hb_peers": [], 2022-03-01T06:36:39.100 INFO:tasks.workunit.client.0.smithi169.stdout: "snap_trim_queue_len": 0, 2022-03-01T06:36:39.101 INFO:tasks.workunit.client.0.smithi169.stdout: "num_snap_trimming": 0, 2022-03-01T06:36:39.101 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shards_repaired": 0, 2022-03-01T06:36:39.101 INFO:tasks.workunit.client.0.smithi169.stdout: "op_queue_age_hist": { 2022-03-01T06:36:39.101 INFO:tasks.workunit.client.0.smithi169.stdout: "histogram": [], 2022-03-01T06:36:39.102 INFO:tasks.workunit.client.0.smithi169.stdout: "upper_bound": 1 2022-03-01T06:36:39.102 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.102 INFO:tasks.workunit.client.0.smithi169.stdout: "perf_stat": { 2022-03-01T06:36:39.102 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ms": 0, 2022-03-01T06:36:39.103 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ms": 0, 2022-03-01T06:36:39.103 INFO:tasks.workunit.client.0.smithi169.stdout: "commit_latency_ns": 0, 2022-03-01T06:36:39.103 INFO:tasks.workunit.client.0.smithi169.stdout: "apply_latency_ns": 0 2022-03-01T06:36:39.103 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.104 INFO:tasks.workunit.client.0.smithi169.stdout: "alerts": [], 2022-03-01T06:36:39.104 INFO:tasks.workunit.client.0.smithi169.stdout: "network_ping_times": [] 2022-03-01T06:36:39.104 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.104 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.104 INFO:tasks.workunit.client.0.smithi169.stdout: "pool_stats": [ 2022-03-01T06:36:39.105 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.105 INFO:tasks.workunit.client.0.smithi169.stdout: "poolid": 1, 2022-03-01T06:36:39.105 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg": 1, 2022-03-01T06:36:39.105 INFO:tasks.workunit.client.0.smithi169.stdout: "stat_sum": { 2022-03-01T06:36:39.106 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes": 47947, 2022-03-01T06:36:39.106 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects": 100, 2022-03-01T06:36:39.106 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_clones": 0, 2022-03-01T06:36:39.106 INFO:tasks.workunit.client.0.smithi169.stdout: "num_object_copies": 300, 2022-03-01T06:36:39.107 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing_on_primary": 0, 2022-03-01T06:36:39.107 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_missing": 0, 2022-03-01T06:36:39.107 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_degraded": 100, 2022-03-01T06:36:39.107 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_misplaced": 0, 2022-03-01T06:36:39.108 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_unfound": 0, 2022-03-01T06:36:39.108 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_dirty": 100, 2022-03-01T06:36:39.108 INFO:tasks.workunit.client.0.smithi169.stdout: "num_whiteouts": 0, 2022-03-01T06:36:39.109 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read": 0, 2022-03-01T06:36:39.109 INFO:tasks.workunit.client.0.smithi169.stdout: "num_read_kb": 0, 2022-03-01T06:36:39.109 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write": 101, 2022-03-01T06:36:39.109 INFO:tasks.workunit.client.0.smithi169.stdout: "num_write_kb": 101, 2022-03-01T06:36:39.110 INFO:tasks.workunit.client.0.smithi169.stdout: "num_scrub_errors": 0, 2022-03-01T06:36:39.110 INFO:tasks.workunit.client.0.smithi169.stdout: "num_shallow_scrub_errors": 0, 2022-03-01T06:36:39.110 INFO:tasks.workunit.client.0.smithi169.stdout: "num_deep_scrub_errors": 0, 2022-03-01T06:36:39.110 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_recovered": 0, 2022-03-01T06:36:39.111 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_recovered": 0, 2022-03-01T06:36:39.111 INFO:tasks.workunit.client.0.smithi169.stdout: "num_keys_recovered": 0, 2022-03-01T06:36:39.111 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_omap": 0, 2022-03-01T06:36:39.112 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_hit_set_archive": 0, 2022-03-01T06:36:39.112 INFO:tasks.workunit.client.0.smithi169.stdout: "num_bytes_hit_set_archive": 0, 2022-03-01T06:36:39.112 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush": 0, 2022-03-01T06:36:39.112 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_kb": 0, 2022-03-01T06:36:39.113 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict": 0, 2022-03-01T06:36:39.113 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_kb": 0, 2022-03-01T06:36:39.113 INFO:tasks.workunit.client.0.smithi169.stdout: "num_promote": 0, 2022-03-01T06:36:39.113 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_high": 0, 2022-03-01T06:36:39.114 INFO:tasks.workunit.client.0.smithi169.stdout: "num_flush_mode_low": 0, 2022-03-01T06:36:39.114 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_some": 0, 2022-03-01T06:36:39.114 INFO:tasks.workunit.client.0.smithi169.stdout: "num_evict_mode_full": 0, 2022-03-01T06:36:39.114 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_pinned": 0, 2022-03-01T06:36:39.115 INFO:tasks.workunit.client.0.smithi169.stdout: "num_legacy_snapsets": 0, 2022-03-01T06:36:39.115 INFO:tasks.workunit.client.0.smithi169.stdout: "num_large_omap_objects": 0, 2022-03-01T06:36:39.115 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_manifest": 0, 2022-03-01T06:36:39.115 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_bytes": 0, 2022-03-01T06:36:39.115 INFO:tasks.workunit.client.0.smithi169.stdout: "num_omap_keys": 0, 2022-03-01T06:36:39.116 INFO:tasks.workunit.client.0.smithi169.stdout: "num_objects_repaired": 0 2022-03-01T06:36:39.116 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.116 INFO:tasks.workunit.client.0.smithi169.stdout: "store_stats": { 2022-03-01T06:36:39.116 INFO:tasks.workunit.client.0.smithi169.stdout: "total": 0, 2022-03-01T06:36:39.117 INFO:tasks.workunit.client.0.smithi169.stdout: "available": 0, 2022-03-01T06:36:39.117 INFO:tasks.workunit.client.0.smithi169.stdout: "internally_reserved": 0, 2022-03-01T06:36:39.117 INFO:tasks.workunit.client.0.smithi169.stdout: "allocated": 19529728, 2022-03-01T06:36:39.117 INFO:tasks.workunit.client.0.smithi169.stdout: "data_stored": 142522, 2022-03-01T06:36:39.117 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed": 0, 2022-03-01T06:36:39.118 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_allocated": 0, 2022-03-01T06:36:39.118 INFO:tasks.workunit.client.0.smithi169.stdout: "data_compressed_original": 0, 2022-03-01T06:36:39.118 INFO:tasks.workunit.client.0.smithi169.stdout: "omap_allocated": 0, 2022-03-01T06:36:39.118 INFO:tasks.workunit.client.0.smithi169.stdout: "internal_metadata": 0 2022-03-01T06:36:39.118 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.119 INFO:tasks.workunit.client.0.smithi169.stdout: "log_size": 17, 2022-03-01T06:36:39.119 INFO:tasks.workunit.client.0.smithi169.stdout: "ondisk_log_size": 17, 2022-03-01T06:36:39.119 INFO:tasks.workunit.client.0.smithi169.stdout: "up": 2, 2022-03-01T06:36:39.119 INFO:tasks.workunit.client.0.smithi169.stdout: "acting": 2, 2022-03-01T06:36:39.120 INFO:tasks.workunit.client.0.smithi169.stdout: "num_store_stats": 3 2022-03-01T06:36:39.120 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.120 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.120 INFO:tasks.workunit.client.0.smithi169.stdout: "osd_stats": [ 2022-03-01T06:36:39.121 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.121 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:39.121 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 146028888083 2022-03-01T06:36:39.121 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.121 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.122 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:39.122 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 42949672977 2022-03-01T06:36:39.122 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.123 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.123 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:39.123 INFO:tasks.workunit.client.0.smithi169.stdout: "seq": 154618822674 2022-03-01T06:36:39.123 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.123 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.124 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_by_state": [ 2022-03-01T06:36:39.124 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.124 INFO:tasks.workunit.client.0.smithi169.stdout: "state": "active+undersized+degraded", 2022-03-01T06:36:39.124 INFO:tasks.workunit.client.0.smithi169.stdout: "num": 1 2022-03-01T06:36:39.125 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.125 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.125 INFO:tasks.workunit.client.0.smithi169.stdout: "num_pg_by_osd": [ 2022-03-01T06:36:39.125 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.125 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 2, 2022-03-01T06:36:39.126 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 0, 2022-03-01T06:36:39.126 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 1, 2022-03-01T06:36:39.126 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:36:39.126 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.127 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.127 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 1, 2022-03-01T06:36:39.127 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 0, 2022-03-01T06:36:39.127 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 0, 2022-03-01T06:36:39.127 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:36:39.128 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.128 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.128 INFO:tasks.workunit.client.0.smithi169.stdout: "osd": 0, 2022-03-01T06:36:39.128 INFO:tasks.workunit.client.0.smithi169.stdout: "num_primary_pg": 1, 2022-03-01T06:36:39.129 INFO:tasks.workunit.client.0.smithi169.stdout: "num_acting_pg": 1, 2022-03-01T06:36:39.129 INFO:tasks.workunit.client.0.smithi169.stdout: "num_up_not_acting_pg": 0 2022-03-01T06:36:39.129 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.129 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.129 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snaps": [ 2022-03-01T06:36:39.130 INFO:tasks.workunit.client.0.smithi169.stdout: { 2022-03-01T06:36:39.130 INFO:tasks.workunit.client.0.smithi169.stdout: "pool": 1, 2022-03-01T06:36:39.130 INFO:tasks.workunit.client.0.smithi169.stdout: "purged_snaps": {} 2022-03-01T06:36:39.130 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.130 INFO:tasks.workunit.client.0.smithi169.stdout: ], 2022-03-01T06:36:39.131 INFO:tasks.workunit.client.0.smithi169.stdout: "servicemap": { 2022-03-01T06:36:39.131 INFO:tasks.workunit.client.0.smithi169.stdout: "epoch": 5, 2022-03-01T06:36:39.131 INFO:tasks.workunit.client.0.smithi169.stdout: "modified": "2022-03-01T06:36:30.152464+0000", 2022-03-01T06:36:39.131 INFO:tasks.workunit.client.0.smithi169.stdout: "services": {} 2022-03-01T06:36:39.132 INFO:tasks.workunit.client.0.smithi169.stdout: }, 2022-03-01T06:36:39.132 INFO:tasks.workunit.client.0.smithi169.stdout: "paxos": { 2022-03-01T06:36:39.132 INFO:tasks.workunit.client.0.smithi169.stdout: "first_committed": 1, 2022-03-01T06:36:39.132 INFO:tasks.workunit.client.0.smithi169.stdout: "last_committed": 418, 2022-03-01T06:36:39.132 INFO:tasks.workunit.client.0.smithi169.stdout: "last_pn": 0, 2022-03-01T06:36:39.133 INFO:tasks.workunit.client.0.smithi169.stdout: "accepted_pn": 0 2022-03-01T06:36:39.133 INFO:tasks.workunit.client.0.smithi169.stdout: } 2022-03-01T06:36:39.133 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:36:39.134 INFO:tasks.workunit.client.0.smithi169.stderr:report 1497873695 2022-03-01T06:36:39.134 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2022-03-01T06:36:39.135 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2022-03-01T06:36:39.135 INFO:tasks.workunit.client.0.smithi169.stdout:delay recovery 2022-03-01T06:36:39.136 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:39.136 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:39.136 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2022-03-01T06:35:31.320549+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:39.137 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:39.137 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:39.137 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:39.139 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:36:39.139 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2022-03-01T06:36:39.139 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-03-01T06:36:39.139 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-03-01T06:36:39.140 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:36:39.140 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:36:39.140 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:36:39.140 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.0.asok 2022-03-01T06:36:39.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2022-03-01T06:36:39.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.0.asok set_recovery_delay 100000 2022-03-01T06:36:39.225 INFO:tasks.workunit.client.0.smithi169.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-03-01T06:36:39.232 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2022-03-01T06:36:39.234 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2022-03-01T06:36:39.234 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-03-01T06:36:39.234 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-03-01T06:36:39.235 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:36:39.235 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:36:39.235 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:36:39.235 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.2.asok 2022-03-01T06:36:39.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2022-03-01T06:36:39.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.2.asok set_recovery_delay 100000 2022-03-01T06:36:39.319 INFO:tasks.workunit.client.0.smithi169.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-03-01T06:36:39.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2022-03-01T06:36:39.327 INFO:tasks.workunit.client.0.smithi169.stdout:revive divergent 1 2022-03-01T06:36:39.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:39.628 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:39.628 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2022-03-01T06:35:31.320549+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:39.628 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:39.629 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:39.629 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:39.638 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2022-03-01T06:36:40.316 INFO:tasks.workunit.client.0.smithi169.stderr:noup is set 2022-03-01T06:36:40.328 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2022-03-01T06:36:40.328 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:36:40.328 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:36:40.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-03-01T06:36:40.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:36:40.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:36:40.329 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:36:40.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:36:40.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:36:40.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:36:40.330 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:36:40.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:36:40.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:36:40.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:36:40.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:36:40.332 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:36:40.332 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:36:40.333 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:36:40.333 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:36:40.333 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:36:40.333 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:36:40.334 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:36:40.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:36:40.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:36:40.334 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:36:40.335 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:36:40.335 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:36:40.335 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:36:40.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:36:40.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:36:40.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:36:40.336 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:36:40.337 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:36:40.338 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-03-01T06:36:40.338 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:36:40.339 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-03-01T06:36:40.339 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-03-01T06:36:40.339 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:36:40.340 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:36:40.340 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:36:40.354 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:40.375+0000 7f87c96a5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:36:40.356 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:40.377+0000 7f87c96a5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:36:40.357 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:40.378+0000 7f87c96a5f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:36:40.660 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2022-03-01T06:36:40.660 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2022-03-01T06:36:40.877 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:40.897+0000 7f87c96a5f00 -1 Falling back to public interface 2022-03-01T06:36:41.717 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:41.738+0000 7f87c96a5f00 -1 osd.1 27 log_to_monitors {default=true} 2022-03-01T06:36:45.661 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2022-03-01T06:36:45.662 INFO:tasks.workunit.client.0.smithi169.stdout:delay recovery divergent 2022-03-01T06:36:45.662 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:45.974 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:45.975 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2022-03-01T06:35:31.320549+0000 39'101 41:124 [0,2] 0 [0,2] 0 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:45.975 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:45.975 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:45.976 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:45.985 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2022-03-01T06:36:45.986 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-03-01T06:36:45.986 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-03-01T06:36:45.986 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:36:45.986 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:36:45.987 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:36:45.987 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.1.asok 2022-03-01T06:36:45.987 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2022-03-01T06:36:45.987 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.1.asok set_recovery_delay 100000 2022-03-01T06:36:46.072 INFO:tasks.workunit.client.0.smithi169.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2022-03-01T06:36:46.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2022-03-01T06:36:46.647 INFO:tasks.workunit.client.0.smithi169.stderr:noup is unset 2022-03-01T06:36:46.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2022-03-01T06:36:46.656 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:36:46.656 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:36:46.656 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:36:46.657 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:36:46.657 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:36:46.657 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:36:46.657 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:36:46.658 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:36:46.658 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:36:47.101 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/44358,v1:127.0.0.1:6803/44358] [v2:127.0.0.1:6804/44358,v1:127.0.0.1:6805/44358] exists,up 1e0fcd1b-9e36-4e3d-b2fa-3b1c69d879a8 2022-03-01T06:36:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:36:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:36:47.103 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:36:47.104 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2022-03-01T06:36:47.104 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:36:47.104 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:36:47.104 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:36:47.104 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:36:47.105 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:36:47.105 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:36:47.106 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:36:47.106 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:36:47.107 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:36:47.428 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/47041,v1:127.0.0.1:6819/47041] [v2:127.0.0.1:6820/47041,v1:127.0.0.1:6821/47041] exists,up 37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:36:47.429 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:36:47.429 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:36:47.430 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:36:47.430 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2022-03-01T06:36:47.430 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:36:47.430 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:36:47.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:36:47.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:36:47.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:36:47.431 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:36:47.432 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:36:47.432 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:36:47.433 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:36:47.749 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/44552,v1:127.0.0.1:6811/44552] [v2:127.0.0.1:6812/44552,v1:127.0.0.1:6813/44552] exists,up dfb1e4b2-84de-4539-b25a-a5ab79b41e6d 2022-03-01T06:36:47.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:36:47.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:36:47.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:36:47.751 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:48.051 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:48.052 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2022-03-01T06:36:46.788547+0000 27'106 44:118 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:48.052 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:48.052 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:48.053 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:48.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2022-03-01T06:36:48.062 INFO:tasks.workunit.client.0.smithi169.stdout:wait for peering 2022-03-01T06:36:48.062 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:48.369 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:48.370 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2022-03-01T06:36:46.788547+0000 27'106 44:118 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:48.370 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:48.370 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:48.371 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:48.378 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2022-03-01T06:36:48.487 INFO:tasks.workunit.client.0.smithi169.stdout:killing divergent 1 2022-03-01T06:36:48.487 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2022-03-01T06:36:48.487 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:48.790 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:48.790 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2022-03-01T06:36:46.788547+0000 27'106 44:118 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:48.790 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:48.791 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:48.791 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:48.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2022-03-01T06:36:48.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:36:48.801 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:36:48.801 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:36:48.801 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:36:48.801 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:36:48.906 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:36:48.907 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.31520.out 2022-03-01T06:36:48.907 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.31520.out 2022-03-01T06:36:48.907 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2022-03-01T06:36:48.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2022-03-01T06:36:48.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2022-03-01T06:36:48.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2022-03-01T06:36:48.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2022-03-01T06:36:48.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.31520.out 2022-03-01T06:36:49.515 INFO:tasks.workunit.client.0.smithi169.stderr:Exporting 1.0 info 1.0( v 45'102 lc 0'0 (27'90,45'102] local-lis/les=44/45 n=101 ec=21/21 lis/c=44/21 les/c/f=45/24/0 sis=44) 2022-03-01T06:36:49.523 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:023dd3d2:::existing_12:head# 2022-03-01T06:36:49.537 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:04c40a0c:::existing_93:head# 2022-03-01T06:36:49.543 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:064ff5fb:::existing_36:head# 2022-03-01T06:36:49.565 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:091f9f1d:::existing_94:head# 2022-03-01T06:36:49.570 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:0a865a93:::existing_54:head# 2022-03-01T06:36:49.573 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:0fed513c:::existing_41:head# 2022-03-01T06:36:49.575 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:13b9e0b8:::existing_92:head# 2022-03-01T06:36:49.584 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:169157fa:::existing_81:head# 2022-03-01T06:36:49.588 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:17731382:::existing_58:head# 2022-03-01T06:36:49.592 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:18a45a39:::existing_43:head# 2022-03-01T06:36:49.592 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:1c160f01:::existing_66:head# 2022-03-01T06:36:49.594 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:1dd6c249:::existing_97:head# 2022-03-01T06:36:49.595 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:227e0c77:::existing_7:head# 2022-03-01T06:36:49.595 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:23e1b063:::existing_59:head# 2022-03-01T06:36:49.596 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:28566edb:::existing_78:head# 2022-03-01T06:36:49.604 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:2a101182:::existing_20:head# 2022-03-01T06:36:49.610 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:2d1519da:::existing_96:head# 2022-03-01T06:36:49.610 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:3078b87a:::existing_86:head# 2022-03-01T06:36:49.611 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:30bde01b:::existing_15:head# 2022-03-01T06:36:49.612 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:317a7e19:::existing_56:head# 2022-03-01T06:36:49.612 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:38b9c7fa:::existing_33:head# 2022-03-01T06:36:49.614 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:3e16d84b:::existing_48:head# 2022-03-01T06:36:49.615 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:45fe8203:::existing_75:head# 2022-03-01T06:36:49.616 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:4e82c9ba:::existing_50:head# 2022-03-01T06:36:49.617 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:5298dcdb:::existing_57:head# 2022-03-01T06:36:49.617 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:52ebaa44:::existing_32:head# 2022-03-01T06:36:49.617 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:53588912:::existing_9:head# 2022-03-01T06:36:49.619 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:5b7e2de6:::existing_84:head# 2022-03-01T06:36:49.620 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:5da0255c:::existing_13:head# 2022-03-01T06:36:49.620 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:5da2dab8:::existing_53:head# 2022-03-01T06:36:49.620 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:5ee3514b:::existing_30:head# 2022-03-01T06:36:49.620 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:602f83fe:::foo:head# 2022-03-01T06:36:49.620 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:60426cc7:::existing_62:head# 2022-03-01T06:36:49.622 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:628918ca:::existing_79:head# 2022-03-01T06:36:49.622 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:63f8ab2d:::existing_77:head# 2022-03-01T06:36:49.622 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:67e1780f:::existing_27:head# 2022-03-01T06:36:49.623 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:68143ca1:::existing_71:head# 2022-03-01T06:36:49.623 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:6bac9509:::existing_29:head# 2022-03-01T06:36:49.624 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:6bbec798:::existing_45:head# 2022-03-01T06:36:49.625 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:6cb23a40:::existing_65:head# 2022-03-01T06:36:49.625 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:7055ec5e:::existing_83:head# 2022-03-01T06:36:49.625 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:70a54e2b:::existing_87:head# 2022-03-01T06:36:49.632 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:750ace21:::existing_19:head# 2022-03-01T06:36:49.633 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:78623b1d:::existing_6:head# 2022-03-01T06:36:49.633 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:79881152:::existing_63:head# 2022-03-01T06:36:49.634 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:7a80c0d5:::existing_99:head# 2022-03-01T06:36:49.636 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:81490677:::existing_76:head# 2022-03-01T06:36:49.637 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:85ec8e5d:::existing_85:head# 2022-03-01T06:36:49.638 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:8e2abbf0:::existing_49:head# 2022-03-01T06:36:49.638 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:8ee0fe0f:::existing_52:head# 2022-03-01T06:36:49.639 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:923c5d4d:::existing_5:head# 2022-03-01T06:36:49.640 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:92731e71:::existing_89:head# 2022-03-01T06:36:49.640 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:928205cf:::existing_82:head# 2022-03-01T06:36:49.641 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:93be2267:::existing_24:head# 2022-03-01T06:36:49.641 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:99ceda2c:::existing_44:head# 2022-03-01T06:36:49.643 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:9dd917ee:::existing_26:head# 2022-03-01T06:36:49.643 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a28805a6:::existing_72:head# 2022-03-01T06:36:49.643 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a294a8f6:::existing_16:head# 2022-03-01T06:36:49.645 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a2f5473a:::existing_42:head# 2022-03-01T06:36:49.645 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a504a533:::existing_90:head# 2022-03-01T06:36:49.645 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a691337e:::existing_61:head# 2022-03-01T06:36:49.646 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a6e8d7ac:::existing_28:head# 2022-03-01T06:36:49.647 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:a9fde86e:::existing_18:head# 2022-03-01T06:36:49.647 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:aa309faa:::existing_68:head# 2022-03-01T06:36:49.647 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:ae40b0ed:::existing_31:head# 2022-03-01T06:36:49.648 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:b279b1fe:::existing_10:head# 2022-03-01T06:36:49.649 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:b343e277:::existing_88:head# 2022-03-01T06:36:49.650 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:b7130e90:::existing_80:head# 2022-03-01T06:36:49.650 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:bc6f7309:::existing_70:head# 2022-03-01T06:36:49.650 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:be8489b7:::existing_38:head# 2022-03-01T06:36:49.651 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:beb4e5e3:::existing_47:head# 2022-03-01T06:36:49.651 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:bebec877:::existing_74:head# 2022-03-01T06:36:49.652 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:c04347d8:::existing_60:head# 2022-03-01T06:36:49.652 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:c156033e:::existing_100:head# 2022-03-01T06:36:49.653 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:c19f5a8e:::existing_8:head# 2022-03-01T06:36:49.653 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:c3e6ebe4:::existing_17:head# 2022-03-01T06:36:49.659 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:c6c5a7c2:::existing_39:head# 2022-03-01T06:36:49.659 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:cc1d0abf:::existing_95:head# 2022-03-01T06:36:49.660 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:ccb3865b:::existing_73:head# 2022-03-01T06:36:49.661 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:cd84e19e:::existing_40:head# 2022-03-01T06:36:49.661 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:d2cd64ce:::existing_37:head# 2022-03-01T06:36:49.661 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:d41a25c9:::existing_22:head# 2022-03-01T06:36:49.662 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:d7f088da:::existing_91:head# 2022-03-01T06:36:49.663 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:d9cbd1d6:::existing_64:head# 2022-03-01T06:36:49.663 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:dcc4ff82:::existing_98:head# 2022-03-01T06:36:49.664 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:dd677bd3:::existing_35:head# 2022-03-01T06:36:49.665 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:dd6d90e7:::existing_46:head# 2022-03-01T06:36:49.666 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:e0e22077:::existing_21:head# 2022-03-01T06:36:49.667 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:e255b710:::existing_55:head# 2022-03-01T06:36:49.667 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:eff1d5db:::existing_14:head# 2022-03-01T06:36:49.668 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:f1b156bf:::existing_23:head# 2022-03-01T06:36:49.668 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:f5ce4199:::existing_34:head# 2022-03-01T06:36:49.669 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:f6809456:::existing_11:head# 2022-03-01T06:36:49.670 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:f699a2a9:::existing_67:head# 2022-03-01T06:36:49.670 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:f9baac43:::existing_25:head# 2022-03-01T06:36:49.670 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:fb76688e:::existing_51:head# 2022-03-01T06:36:49.671 INFO:tasks.workunit.client.0.smithi169.stderr:Read #1:fbb92eec:::existing_69:head# 2022-03-01T06:36:49.671 INFO:tasks.workunit.client.0.smithi169.stderr:Export successful 2022-03-01T06:36:49.672 INFO:tasks.workunit.client.0.smithi169.stdout: marking collection for removal 2022-03-01T06:36:49.672 INFO:tasks.workunit.client.0.smithi169.stdout:setting '_remove' omap key 2022-03-01T06:36:49.672 INFO:tasks.workunit.client.0.smithi169.stdout:finish_remove_pgs 1.0_head removing 1.0 2022-03-01T06:36:49.904 INFO:tasks.workunit.client.0.smithi169.stderr:Remove successful 2022-03-01T06:36:50.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.31520.out 2022-03-01T06:36:50.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2022-03-01T06:36:50.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2022-03-01T06:36:50.190 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2022-03-01T06:36:50.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2022-03-01T06:36:50.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2022-03-01T06:36:50.191 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.31520.out 2022-03-01T06:36:50.648 INFO:tasks.workunit.client.0.smithi169.stdout:get_pg_num_history pg_num_history pg_num_history(e45 pg_nums {1={21=1}} deleted_pools ) 2022-03-01T06:36:50.649 INFO:tasks.workunit.client.0.smithi169.stdout:Importing pgid 1.0 2022-03-01T06:36:50.649 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:023dd3d2:::existing_12:head# 2022-03-01T06:36:50.649 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.697 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:04c40a0c:::existing_93:head# 2022-03-01T06:36:50.697 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.722 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:064ff5fb:::existing_36:head# 2022-03-01T06:36:50.722 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.747 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:091f9f1d:::existing_94:head# 2022-03-01T06:36:50.748 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.772 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:0a865a93:::existing_54:head# 2022-03-01T06:36:50.773 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.797 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:0fed513c:::existing_41:head# 2022-03-01T06:36:50.798 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.822 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:13b9e0b8:::existing_92:head# 2022-03-01T06:36:50.823 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.848 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:169157fa:::existing_81:head# 2022-03-01T06:36:50.848 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.873 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:17731382:::existing_58:head# 2022-03-01T06:36:50.873 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.931 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:18a45a39:::existing_43:head# 2022-03-01T06:36:50.932 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:50.982 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:1c160f01:::existing_66:head# 2022-03-01T06:36:50.982 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.007 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:1dd6c249:::existing_97:head# 2022-03-01T06:36:51.007 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.032 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:227e0c77:::existing_7:head# 2022-03-01T06:36:51.032 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.057 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:23e1b063:::existing_59:head# 2022-03-01T06:36:51.057 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.090 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:28566edb:::existing_78:head# 2022-03-01T06:36:51.091 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.149 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:2a101182:::existing_20:head# 2022-03-01T06:36:51.149 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.174 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:2d1519da:::existing_96:head# 2022-03-01T06:36:51.174 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.199 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:3078b87a:::existing_86:head# 2022-03-01T06:36:51.200 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.224 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:30bde01b:::existing_15:head# 2022-03-01T06:36:51.225 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.258 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:317a7e19:::existing_56:head# 2022-03-01T06:36:51.258 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.291 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:38b9c7fa:::existing_33:head# 2022-03-01T06:36:51.291 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.325 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:3e16d84b:::existing_48:head# 2022-03-01T06:36:51.325 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.358 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:45fe8203:::existing_75:head# 2022-03-01T06:36:51.359 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.392 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:4e82c9ba:::existing_50:head# 2022-03-01T06:36:51.392 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.425 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:5298dcdb:::existing_57:head# 2022-03-01T06:36:51.426 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.459 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:52ebaa44:::existing_32:head# 2022-03-01T06:36:51.459 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.492 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:53588912:::existing_9:head# 2022-03-01T06:36:51.492 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.525 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:5b7e2de6:::existing_84:head# 2022-03-01T06:36:51.526 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.559 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:5da0255c:::existing_13:head# 2022-03-01T06:36:51.559 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.584 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:5da2dab8:::existing_53:head# 2022-03-01T06:36:51.585 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.610 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:5ee3514b:::existing_30:head# 2022-03-01T06:36:51.610 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.634 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:602f83fe:::foo:head# 2022-03-01T06:36:51.635 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.660 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:60426cc7:::existing_62:head# 2022-03-01T06:36:51.660 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.693 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:628918ca:::existing_79:head# 2022-03-01T06:36:51.694 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.718 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:63f8ab2d:::existing_77:head# 2022-03-01T06:36:51.719 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.764 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:67e1780f:::existing_27:head# 2022-03-01T06:36:51.764 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.794 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:68143ca1:::existing_71:head# 2022-03-01T06:36:51.794 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.819 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:6bac9509:::existing_29:head# 2022-03-01T06:36:51.819 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.844 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:6bbec798:::existing_45:head# 2022-03-01T06:36:51.844 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.869 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:6cb23a40:::existing_65:head# 2022-03-01T06:36:51.869 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.894 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:7055ec5e:::existing_83:head# 2022-03-01T06:36:51.894 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.928 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:70a54e2b:::existing_87:head# 2022-03-01T06:36:51.928 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.961 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:750ace21:::existing_19:head# 2022-03-01T06:36:51.961 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:51.994 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:78623b1d:::existing_6:head# 2022-03-01T06:36:51.995 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.029 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:79881152:::existing_63:head# 2022-03-01T06:36:52.030 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.061 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:7a80c0d5:::existing_99:head# 2022-03-01T06:36:52.062 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.095 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:81490677:::existing_76:head# 2022-03-01T06:36:52.096 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.128 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:85ec8e5d:::existing_85:head# 2022-03-01T06:36:52.128 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.162 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:8e2abbf0:::existing_49:head# 2022-03-01T06:36:52.162 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.206 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:8ee0fe0f:::existing_52:head# 2022-03-01T06:36:52.206 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.245 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:923c5d4d:::existing_5:head# 2022-03-01T06:36:52.246 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.279 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:92731e71:::existing_89:head# 2022-03-01T06:36:52.279 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.312 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:928205cf:::existing_82:head# 2022-03-01T06:36:52.312 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.346 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:93be2267:::existing_24:head# 2022-03-01T06:36:52.346 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.379 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:99ceda2c:::existing_44:head# 2022-03-01T06:36:52.380 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.446 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:9dd917ee:::existing_26:head# 2022-03-01T06:36:52.447 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.480 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a28805a6:::existing_72:head# 2022-03-01T06:36:52.480 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.513 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a294a8f6:::existing_16:head# 2022-03-01T06:36:52.513 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.546 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a2f5473a:::existing_42:head# 2022-03-01T06:36:52.547 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.580 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a504a533:::existing_90:head# 2022-03-01T06:36:52.580 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.614 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a691337e:::existing_61:head# 2022-03-01T06:36:52.614 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.647 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a6e8d7ac:::existing_28:head# 2022-03-01T06:36:52.647 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.680 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:a9fde86e:::existing_18:head# 2022-03-01T06:36:52.681 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.722 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:aa309faa:::existing_68:head# 2022-03-01T06:36:52.723 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.747 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:ae40b0ed:::existing_31:head# 2022-03-01T06:36:52.747 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.839 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:b279b1fe:::existing_10:head# 2022-03-01T06:36:52.839 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.864 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:b343e277:::existing_88:head# 2022-03-01T06:36:52.864 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.898 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:b7130e90:::existing_80:head# 2022-03-01T06:36:52.898 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.948 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:bc6f7309:::existing_70:head# 2022-03-01T06:36:52.948 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.973 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:be8489b7:::existing_38:head# 2022-03-01T06:36:52.973 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:52.998 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:beb4e5e3:::existing_47:head# 2022-03-01T06:36:52.998 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.045 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:bebec877:::existing_74:head# 2022-03-01T06:36:53.045 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.073 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:c04347d8:::existing_60:head# 2022-03-01T06:36:53.074 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.098 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:c156033e:::existing_100:head# 2022-03-01T06:36:53.099 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.123 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:c19f5a8e:::existing_8:head# 2022-03-01T06:36:53.124 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.148 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:c3e6ebe4:::existing_17:head# 2022-03-01T06:36:53.149 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.174 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:c6c5a7c2:::existing_39:head# 2022-03-01T06:36:53.174 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.199 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:cc1d0abf:::existing_95:head# 2022-03-01T06:36:53.199 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.224 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:ccb3865b:::existing_73:head# 2022-03-01T06:36:53.224 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.249 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:cd84e19e:::existing_40:head# 2022-03-01T06:36:53.249 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.274 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:d2cd64ce:::existing_37:head# 2022-03-01T06:36:53.275 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.299 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:d41a25c9:::existing_22:head# 2022-03-01T06:36:53.300 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.325 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:d7f088da:::existing_91:head# 2022-03-01T06:36:53.325 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.350 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:d9cbd1d6:::existing_64:head# 2022-03-01T06:36:53.350 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.375 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:dcc4ff82:::existing_98:head# 2022-03-01T06:36:53.376 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.400 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:dd677bd3:::existing_35:head# 2022-03-01T06:36:53.400 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.425 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:dd6d90e7:::existing_46:head# 2022-03-01T06:36:53.425 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.450 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:e0e22077:::existing_21:head# 2022-03-01T06:36:53.451 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.475 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:e255b710:::existing_55:head# 2022-03-01T06:36:53.476 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.502 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:eff1d5db:::existing_14:head# 2022-03-01T06:36:53.502 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.526 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:f1b156bf:::existing_23:head# 2022-03-01T06:36:53.526 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.551 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:f5ce4199:::existing_34:head# 2022-03-01T06:36:53.551 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.576 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:f6809456:::existing_11:head# 2022-03-01T06:36:53.576 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.601 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:f699a2a9:::existing_67:head# 2022-03-01T06:36:53.601 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.626 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:f9baac43:::existing_25:head# 2022-03-01T06:36:53.626 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.682 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:fb76688e:::existing_51:head# 2022-03-01T06:36:53.682 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.710 INFO:tasks.workunit.client.0.smithi169.stdout:Write #1:fbb92eec:::existing_69:head# 2022-03-01T06:36:53.711 INFO:tasks.workunit.client.0.smithi169.stdout:snapset 0=[]:{} 2022-03-01T06:36:53.735 INFO:tasks.workunit.client.0.smithi169.stdout:write_pg epoch 45 info 1.0( v 45'102 lc 0'0 (27'90,45'102] local-lis/les=44/45 n=101 ec=21/21 lis/c=44/21 les/c/f=45/24/0 sis=44) 2022-03-01T06:36:53.768 INFO:tasks.workunit.client.0.smithi169.stdout:Import successful 2022-03-01T06:36:54.290 INFO:tasks.workunit.client.0.smithi169.stdout:reviving divergent 1 2022-03-01T06:36:54.291 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2022-03-01T06:36:54.291 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:36:54.604 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:36:54.605 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2022-03-01T06:36:46.788547+0000 27'106 44:118 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:36:54.605 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:36:54.605 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:36:54.606 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:36:54.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2022-03-01T06:36:54.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:36:54.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:36:54.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2022-03-01T06:36:54.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:36:54.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:36:54.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:36:54.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:36:54.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:36:54.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:36:54.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:36:54.621 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:36:54.621 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:36:54.621 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:36:54.621 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:36:54.622 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:36:54.622 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:36:54.622 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:36:54.622 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:36:54.623 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:36:54.623 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:36:54.623 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:36:54.623 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:36:54.624 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:36:54.624 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:36:54.624 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:36:54.624 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:36:54.625 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:36:54.625 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:36:54.625 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:36:54.625 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:36:54.626 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:36:54.626 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2022-03-01T06:36:54.626 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:36:54.627 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:36:54.628 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2022-03-01T06:36:54.628 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2022-03-01T06:36:54.628 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:36:54.629 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:36:54.629 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:36:54.645 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:54.665+0000 7f3077047f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:36:54.646 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:54.667+0000 7f3077047f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:36:54.647 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:54.668+0000 7f3077047f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:36:54.950 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2022-03-01T06:36:54.950 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:36:54.950 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:36:54.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:36:54.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:36:54.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:36:54.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:36:54.952 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:36:54.952 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:36:54.952 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:36:55.163 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:55.184+0000 7f3077047f00 -1 Falling back to public interface 2022-03-01T06:36:55.277 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/47041,v1:127.0.0.1:6819/47041] [v2:127.0.0.1:6820/47041,v1:127.0.0.1:6821/47041] exists,up 37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:36:55.278 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:36:55.278 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:36:55.278 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:36:55.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2022-03-01T06:36:55.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:36:55.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:36:55.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:36:55.280 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:36:55.280 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:36:55.280 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:36:55.281 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:36:55.281 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:36:55.282 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:36:55.619 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/47041,v1:127.0.0.1:6819/47041] [v2:127.0.0.1:6820/47041,v1:127.0.0.1:6821/47041] exists,up 37a8e729-ccba-40e7-90ef-c122436009c4 2022-03-01T06:36:55.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:36:55.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:36:55.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:36:55.620 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2022-03-01T06:36:55.940 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:36:55.960+0000 7f3077047f00 -1 osd.1 45 log_to_monitors {default=true} 2022-03-01T06:37:15.621 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2022-03-01T06:37:15.622 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2022-03-01T06:37:15.622 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2022-03-01T06:37:15.622 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:37:15.623 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:15.623 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:15.623 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.1.asok 2022-03-01T06:37:15.623 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2022-03-01T06:37:15.624 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.1.asok dump_ops_in_flight 2022-03-01T06:37:15.713 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:37:15.714 INFO:tasks.workunit.client.0.smithi169.stdout: "ops": [], 2022-03-01T06:37:15.714 INFO:tasks.workunit.client.0.smithi169.stdout: "num_ops": 0 2022-03-01T06:37:15.714 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:37:15.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2022-03-01T06:37:15.722 INFO:tasks.workunit.client.0.smithi169.stdout:allowing recovery 2022-03-01T06:37:15.722 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:37:16.025 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:37:16.026 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2022-03-01T06:36:57.816732+0000 45'102 49:148 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:37:16.026 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:37:16.027 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:37:16.028 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:37:16.035 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2022-03-01T06:37:16.036 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2022-03-01T06:37:16.151 INFO:tasks.workunit.client.0.smithi169.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-03-01T06:37:16.158 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2022-03-01T06:37:16.158 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2022-03-01T06:37:16.275 INFO:tasks.workunit.client.0.smithi169.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-03-01T06:37:16.282 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2022-03-01T06:37:16.283 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2022-03-01T06:37:16.397 INFO:tasks.workunit.client.0.smithi169.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2022-03-01T06:37:16.405 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2022-03-01T06:37:16.405 INFO:tasks.workunit.client.0.smithi169.stdout:reading divergent objects 2022-03-01T06:37:16.406 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2022-03-01T06:37:16.705 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:37:16.706 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2022-03-01T06:36:57.816732+0000 45'102 49:148 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:34:44.852920+0000 0'0 2022-03-01T06:34:44.852920+0000 0 2022-03-01T06:37:16.706 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:37:16.706 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:37:16.707 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:37:16.716 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2022-03-01T06:37:16.717 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2022-03-01T06:37:16.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:37:16.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2022-03-01T06:37:16.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:37:16.747 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2022-03-01T06:37:16.777 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:37:16.778 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2022-03-01T06:37:16.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2022-03-01T06:37:16.810 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2022-03-01T06:37:16.834 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2022-03-01T06:37:16.835 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-03-01T06:37:16.835 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2022-03-01T06:37:16.863 INFO:tasks.workunit.client.0.smithi169.stderr:error getting test/newobject_1: (2) No such file or directory 2022-03-01T06:37:16.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2022-03-01T06:37:16.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2022-03-01T06:37:16.937 INFO:tasks.workunit.client.0.smithi169.stderr:error getting test/newobject_2: (2) No such file or directory 2022-03-01T06:37:16.939 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2022-03-01T06:37:16.940 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2022-03-01T06:37:16.941 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2022-03-01T06:37:16.943 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 27'105 (27'4) delete 1:70b3bf12:::existing_4:head by client.4540.0:1 2022-03-01T06:35:12.612903+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2022-03-01T06:37:16.944 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: keeping 27'105 (27'4) delete 1:70b3bf12:::existing_4:head by client.4540.0:1 2022-03-01T06:35:12.612903+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2022-03-01T06:37:16.944 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-03-01T06:37:16.944 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 27'4 first_divergent_update: 27'105 last_divergent_update: 27'105 2022-03-01T06:37:16.945 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (27'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2022-03-01T06:36:05.024974+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2022-03-01T06:37:16.945 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 27'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2022-03-01T06:35:12.609672+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.945 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: keeping 27'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2022-03-01T06:35:12.609672+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.945 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-03-01T06:37:16.946 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 27'104 last_divergent_update: 27'104 2022-03-01T06:37:16.946 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2022-03-01T06:37:16.946 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2022-03-01T06:37:16.946 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 27'102 (27'3) delete 1:8881e6d8:::existing_3:head by client.4531.0:1 2022-03-01T06:35:12.604326+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2022-03-01T06:37:16.947 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: keeping 27'102 (27'3) delete 1:8881e6d8:::existing_3:head by client.4531.0:1 2022-03-01T06:35:12.604326+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2022-03-01T06:37:16.947 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2022-03-01T06:37:16.947 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 27'3 first_divergent_update: 27'102 last_divergent_update: 27'102 2022-03-01T06:37:16.947 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2022-03-01T06:37:16.947 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2022-03-01T06:37:16.948 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 27'106 2022-03-01T06:37:16.948 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 27'102 (27'3) delete 1:8881e6d8:::existing_3:head by client.4531.0:1 2022-03-01T06:35:12.604326+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2022-03-01T06:37:16.948 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2022-03-01T06:37:16.948 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 27'3 <= info.log_tail 27'90 2022-03-01T06:37:16.949 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 27'106 (27'2) modify 1:97db2a0f:::existing_2:head by client.4541.0:1 2022-03-01T06:35:12.613620+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.949 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: keeping 27'106 (27'2) modify 1:97db2a0f:::existing_2:head by client.4541.0:1 2022-03-01T06:35:12.613620+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.949 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-03-01T06:37:16.949 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 27'2 first_divergent_update: 27'106 last_divergent_update: 27'106 2022-03-01T06:37:16.949 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2022-03-01T06:37:16.950 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2022-03-01T06:37:16.950 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 27'106 2022-03-01T06:37:16.950 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 27'106 (27'2) modify 1:97db2a0f:::existing_2:head by client.4541.0:1 2022-03-01T06:35:12.613620+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.950 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2022-03-01T06:37:16.951 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 27'2 <= info.log_tail 27'90 2022-03-01T06:37:16.951 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 27'101 (27'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2022-03-01T06:35:12.599228+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.951 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: keeping 27'101 (27'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2022-03-01T06:35:12.599228+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.951 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-03-01T06:37:16.952 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 27'1 first_divergent_update: 27'101 last_divergent_update: 27'101 2022-03-01T06:37:16.952 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2022-03-01T06:37:16.952 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2022-03-01T06:37:16.953 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 27'106 2022-03-01T06:37:16.953 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 27'101 (27'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2022-03-01T06:35:12.599228+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.953 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2022-03-01T06:37:16.953 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 27'1 <= info.log_tail 27'90 2022-03-01T06:37:16.954 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 27'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4536.0:1 2022-03-01T06:35:12.608941+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.954 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 20 _merge_object_divergent_entries: keeping 27'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4536.0:1 2022-03-01T06:35:12.608941+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2022-03-01T06:37:16.954 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2022-03-01T06:37:16.954 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 27'103 last_divergent_update: 27'103 2022-03-01T06:37:16.955 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2022-03-01T06:37:16.955 INFO:tasks.workunit.client.0.smithi169.stdout:td/divergent-priors/osd.1.log:2022-03-01T06:36:46.883+0000 7f87a7576700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2022-03-01T06:37:16.956 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2022-03-01T06:37:16.956 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2022-03-01T06:37:16.956 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2022-03-01T06:37:16.956 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2022-03-01T06:37:16.957 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2022-03-01T06:37:16.957 INFO:tasks.workunit.client.0.smithi169.stdout:success 2022-03-01T06:37:16.958 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.31520.out 2022-03-01T06:37:16.959 INFO:tasks.workunit.client.0.smithi169.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.31520.out': No such file or directory 2022-03-01T06:37:16.959 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2022-03-01T06:37:16.959 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2022-03-01T06:37:16.959 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2022-03-01T06:37:17.397 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'test' does not exist 2022-03-01T06:37:17.409 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2022-03-01T06:37:17.410 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:37:17.410 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:37:17.410 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:37:17.411 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:37:17.411 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:37:17.514 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:37:17.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2022-03-01T06:37:17.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-03-01T06:37:17.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:37:17.515 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-03-01T06:37:17.516 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:37:17.516 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:37:17.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:37:17.516 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:37:17.517 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:37:17.519 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:37:17.520 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:37:17.520 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:37:17.521 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:37:17.522 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:37:17.522 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:37:17.522 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:37:17.523 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:37:17.523 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:37:17.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:37:17.524 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:37:17.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:37:17.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:37:17.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-03-01T06:37:17.539 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:37:17.539 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:17.539 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:17.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31520 2022-03-01T06:37:17.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:37:17.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:37:17.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2022-03-01T06:37:17.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2022-03-01T06:37:17.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2022-03-01T06:37:17.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2022-03-01T06:37:17.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-03-01T06:37:17.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2022-03-01T06:37:17.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-03-01T06:37:17.544 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:37:17.544 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:37:17.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:37:17.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:37:17.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:37:17.547 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:37:17.548 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T06:37:17.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T06:37:17.548 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T06:37:17.548 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T06:37:17.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T06:37:17.549 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T06:37:17.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:37:17.549 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T06:37:17.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T06:37:17.550 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T06:37:17.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T06:37:17.552 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2022-03-01T06:37:17.552 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-03-01T06:37:17.552 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T06:37:17.552 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:17.553 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:17.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31520 2022-03-01T06:37:17.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2022-03-01T06:37:17.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2022-03-01T06:37:17.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2022-03-01T06:37:17.554 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2022-03-01T06:37:17.554 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:17.555 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:17.555 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.31520 2022-03-01T06:37:17.558 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2022-03-01T06:37:17.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2022-03-01T06:37:17.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2022-03-01T06:37:17.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2022-03-01T06:37:17.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2022-03-01T06:37:17.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2022-03-01T06:37:17.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2022-03-01T06:37:17.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2022-03-01T06:37:17.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2022-03-01T06:37:17.560 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2022-03-01T06:37:17.560 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2022-03-01T06:37:17.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2022-03-01T06:37:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:37:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:37:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2022-03-01T06:37:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2022-03-01T06:37:17.562 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2022-03-01T06:37:17.562 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2022-03-01T06:37:17.562 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2022-03-01T06:37:17.562 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2022-03-01T06:37:17.563 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2022-03-01T06:37:17.855 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2022-03-01T06:37:17.856 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:37:17.856 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:37:17.856 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:37:17.856 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:17.857 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:17.857 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:17.857 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2022-03-01T06:37:18.014 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2022-03-01T06:37:18.016 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2022-03-01T06:37:18.016 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:37:18.017 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:37:18.017 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2022-03-01T06:37:18.017 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2022-03-01T06:37:18.017 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:37:18.018 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:37:18.018 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:37:18.018 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:37:18.018 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:18.019 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:18.019 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-mon.a.asok 2022-03-01T06:37:18.019 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:37:18.019 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.31520/ceph-mon.a.asok config get fsid 2022-03-01T06:37:18.101 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2022-03-01T06:37:18.102 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2022-03-01T06:37:18.102 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2022-03-01T06:37:18.102 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2022-03-01T06:37:18.102 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2022-03-01T06:37:18.103 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2022-03-01T06:37:18.103 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2022-03-01T06:37:18.103 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2022-03-01T06:37:18.103 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:37:18.103 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:18.104 INFO:tasks.workunit.client.0.smithi169.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:18.104 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-mon.a.asok 2022-03-01T06:37:18.104 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2022-03-01T06:37:18.104 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.31520/ceph-mon.a.asok config get mon_host 2022-03-01T06:37:18.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2022-03-01T06:37:18.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2022-03-01T06:37:18.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2022-03-01T06:37:18.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2022-03-01T06:37:18.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2022-03-01T06:37:18.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2022-03-01T06:37:18.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2022-03-01T06:37:19.039 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2022-03-01T06:37:19.040 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:37:19.040 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:37:19.041 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:37:19.041 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:19.042 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:19.042 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:19.043 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:37:19.043 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2022-03-01T06:37:19.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2022-03-01T06:37:19.069 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2022-03-01T06:37:19.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:37:19.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:37:19.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2022-03-01T06:37:19.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:37:19.071 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2022-03-01T06:37:19.071 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:37:19.071 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:37:19.072 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:37:19.072 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:37:19.072 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-03-01T06:37:19.073 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-03-01T06:37:19.073 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:37:19.073 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:37:19.073 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:37:19.074 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:37:19.074 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:37:19.074 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:37:19.075 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:37:19.075 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:19.075 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:19.075 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:19.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:19.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:37:19.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:37:19.076 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:37:19.077 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:37:19.077 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:37:19.078 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:37:19.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:37:19.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:37:19.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:37:19.079 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:37:19.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2022-03-01T06:37:19.080 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:37:19.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e1cd9c7c-2111-4f83-aa80-69209d7cbd2f 2022-03-01T06:37:19.081 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 e1cd9c7c-2111-4f83-aa80-69209d7cbd2f' 2022-03-01T06:37:19.081 INFO:tasks.workunit.client.0.smithi169.stdout:add osd0 e1cd9c7c-2111-4f83-aa80-69209d7cbd2f 2022-03-01T06:37:19.082 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:37:19.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAfvx1iZwtkBhAAgROYvfQ68xp30mZJ/Al5uQ== 2022-03-01T06:37:19.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAfvx1iZwtkBhAAgROYvfQ68xp30mZJ/Al5uQ=="}' 2022-03-01T06:37:19.087 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e1cd9c7c-2111-4f83-aa80-69209d7cbd2f -i td/divergent-priors/0/new.json 2022-03-01T06:37:19.411 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:37:19.421 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2022-03-01T06:37:19.422 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAfvx1iZwtkBhAAgROYvfQ68xp30mZJ/Al5uQ== --osd-uuid e1cd9c7c-2111-4f83-aa80-69209d7cbd2f 2022-03-01T06:37:19.451 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.472+0000 7f207daedf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:19.453 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.473+0000 7f207daedf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:19.453 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.474+0000 7f207daedf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:19.454 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.475+0000 7f207daedf00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:37:19.455 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.475+0000 7f207daedf00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:37:19.455 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.475+0000 7f207daedf00 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2022-03-01T06:37:19.496 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.516+0000 7f207daedf00 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2022-03-01T06:37:19.856 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:19.877+0000 7f207daedf00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:37:22.494 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2022-03-01T06:37:22.494 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:37:22.495 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2022-03-01T06:37:22.495 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd0 key to auth repository 2022-03-01T06:37:22.496 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:37:22.932 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2022-03-01T06:37:22.933 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:37:22.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:37:22.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:37:22.936 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:37:22.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:37:22.961 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:22.981+0000 7f01c3eebf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:22.962 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:22.983+0000 7f01c3eebf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:22.963 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:22.984+0000 7f01c3eebf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:23.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2022-03-01T06:37:23.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:37:23.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:37:23.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:37:23.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:37:23.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:23.310 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:37:23.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:37:23.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:23.311 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:37:23.484 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:23.504+0000 7f01c3eebf00 -1 Falling back to public interface 2022-03-01T06:37:23.634 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:37:24.202 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:24.222+0000 7f01c3eebf00 -1 osd.0 0 log_to_monitors {default=true} 2022-03-01T06:37:24.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:37:24.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:24.636 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:37:24.637 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:37:24.638 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:24.639 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:37:25.006 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/48526,v1:127.0.0.1:6803/48526] [v2:127.0.0.1:6804/48526,v1:127.0.0.1:6805/48526] exists,up e1cd9c7c-2111-4f83-aa80-69209d7cbd2f 2022-03-01T06:37:25.017 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:37:25.018 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:37:25.018 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:37:25.019 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2022-03-01T06:37:25.019 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2022-03-01T06:37:25.019 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:37:25.020 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:37:25.021 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2022-03-01T06:37:25.021 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:37:25.022 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2022-03-01T06:37:25.022 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:37:25.022 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:37:25.023 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:37:25.023 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:37:25.024 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2022-03-01T06:37:25.024 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2022-03-01T06:37:25.025 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:37:25.025 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:37:25.025 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:37:25.025 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:37:25.026 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:37:25.026 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:37:25.026 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:37:25.027 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:25.027 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:25.027 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:25.027 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:25.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:37:25.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:37:25.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:37:25.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:37:25.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:37:25.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:37:25.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:37:25.030 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:37:25.030 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:37:25.030 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:37:25.031 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2022-03-01T06:37:25.031 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:37:25.031 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=51548123-a49f-434a-911a-0ffa87a2188d 2022-03-01T06:37:25.032 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 51548123-a49f-434a-911a-0ffa87a2188d' 2022-03-01T06:37:25.033 INFO:tasks.workunit.client.0.smithi169.stdout:add osd1 51548123-a49f-434a-911a-0ffa87a2188d 2022-03-01T06:37:25.034 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:37:25.034 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAlvx1iOBb7AhAART2ZrQYnCCKmXsZiTwk+TQ== 2022-03-01T06:37:25.035 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAlvx1iOBb7AhAART2ZrQYnCCKmXsZiTwk+TQ=="}' 2022-03-01T06:37:25.035 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 51548123-a49f-434a-911a-0ffa87a2188d -i td/divergent-priors/1/new.json 2022-03-01T06:37:25.528 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:37:25.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2022-03-01T06:37:25.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlvx1iOBb7AhAART2ZrQYnCCKmXsZiTwk+TQ== --osd-uuid 51548123-a49f-434a-911a-0ffa87a2188d 2022-03-01T06:37:25.568 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.589+0000 7f13daf25f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:25.570 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.591+0000 7f13daf25f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:25.571 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.592+0000 7f13daf25f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:25.572 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.592+0000 7f13daf25f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:37:25.572 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.592+0000 7f13daf25f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:37:25.572 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.592+0000 7f13daf25f00 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2022-03-01T06:37:25.618 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:25.639+0000 7f13daf25f00 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2022-03-01T06:37:25.979 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:26.000+0000 7f13daf25f00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:37:28.689 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2022-03-01T06:37:28.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:37:28.690 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2022-03-01T06:37:28.691 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd1 key to auth repository 2022-03-01T06:37:28.691 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:37:29.125 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.1 2022-03-01T06:37:29.126 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2022-03-01T06:37:29.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:37:29.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:37:29.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:37:29.127 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:37:29.154 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:29.174+0000 7f3032151f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:29.157 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:29.176+0000 7f3032151f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:29.157 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:29.177+0000 7f3032151f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:29.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2022-03-01T06:37:29.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:37:29.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2022-03-01T06:37:29.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:37:29.472 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:37:29.472 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:29.472 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:37:29.473 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:37:29.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:29.473 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:37:29.672 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:29.693+0000 7f3032151f00 -1 Falling back to public interface 2022-03-01T06:37:29.798 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:37:30.430 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:30.450+0000 7f3032151f00 -1 osd.1 0 log_to_monitors {default=true} 2022-03-01T06:37:30.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:37:30.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:30.800 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:37:30.800 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:37:30.801 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:30.801 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2022-03-01T06:37:31.154 INFO:tasks.workunit.client.0.smithi169.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/48881,v1:127.0.0.1:6811/48881] [v2:127.0.0.1:6812/48881,v1:127.0.0.1:6813/48881] exists,up 51548123-a49f-434a-911a-0ffa87a2188d 2022-03-01T06:37:31.159 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:37:31.159 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:37:31.159 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:37:31.160 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2022-03-01T06:37:31.160 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2022-03-01T06:37:31.160 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2022-03-01T06:37:31.160 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2022-03-01T06:37:31.160 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2022-03-01T06:37:31.161 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2022-03-01T06:37:31.161 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2022-03-01T06:37:31.161 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:37:31.162 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:37:31.163 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:37:31.163 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:37:31.163 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-03-01T06:37:31.163 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-03-01T06:37:31.164 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2022-03-01T06:37:31.164 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2022-03-01T06:37:31.164 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:37:31.164 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2022-03-01T06:37:31.165 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:37:31.165 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:37:31.165 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:37:31.165 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:37:31.166 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:37:31.166 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:31.166 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:37:31.166 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:37:31.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2022-03-01T06:37:31.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2022-03-01T06:37:31.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:37:31.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:37:31.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:37:31.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:37:31.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:37:31.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2022-03-01T06:37:31.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2022-03-01T06:37:31.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2022-03-01T06:37:31.169 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2022-03-01T06:37:31.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8819d842-0ff0-4da3-9a3e-6344e0c5f1d5 2022-03-01T06:37:31.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8819d842-0ff0-4da3-9a3e-6344e0c5f1d5' 2022-03-01T06:37:31.171 INFO:tasks.workunit.client.0.smithi169.stdout:add osd2 8819d842-0ff0-4da3-9a3e-6344e0c5f1d5 2022-03-01T06:37:31.171 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2022-03-01T06:37:31.177 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQArvx1idW/JCxAAgAzWOoiVBZ39W9M+gIXvdQ== 2022-03-01T06:37:31.177 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQArvx1idW/JCxAAgAzWOoiVBZ39W9M+gIXvdQ=="}' 2022-03-01T06:37:31.177 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8819d842-0ff0-4da3-9a3e-6344e0c5f1d5 -i td/divergent-priors/2/new.json 2022-03-01T06:37:31.687 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:37:31.698 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2022-03-01T06:37:31.698 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQArvx1idW/JCxAAgAzWOoiVBZ39W9M+gIXvdQ== --osd-uuid 8819d842-0ff0-4da3-9a3e-6344e0c5f1d5 2022-03-01T06:37:31.729 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.750+0000 7fef52bdaf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:31.730 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.751+0000 7fef52bdaf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:31.731 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.752+0000 7fef52bdaf00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:31.732 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.753+0000 7fef52bdaf00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:37:31.732 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.753+0000 7fef52bdaf00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:37:31.732 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.753+0000 7fef52bdaf00 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2022-03-01T06:37:31.795 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:31.816+0000 7fef52bdaf00 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2022-03-01T06:37:32.156 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:32.177+0000 7fef52bdaf00 -1 freelist read_size_meta_from_db missing size meta in DB 2022-03-01T06:37:34.805 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2022-03-01T06:37:34.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2022-03-01T06:37:34.806 INFO:tasks.workunit.client.0.smithi169.stdout:adding osd2 key to auth repository 2022-03-01T06:37:34.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2022-03-01T06:37:34.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2022-03-01T06:37:35.245 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2022-03-01T06:37:35.245 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:37:35.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2022-03-01T06:37:35.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:37:35.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2022-03-01T06:37:35.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2022-03-01T06:37:35.271 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:35.292+0000 7f7fc8f06f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:35.289 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:35.310+0000 7f7fc8f06f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:35.290 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:35.311+0000 7f7fc8f06f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:37:35.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2022-03-01T06:37:35.576 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:37:35.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:37:35.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:37:35.578 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:37:35.578 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:35.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:37:35.579 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:37:35.580 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:35.581 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:37:35.805 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:35.826+0000 7f7fc8f06f00 -1 Falling back to public interface 2022-03-01T06:37:35.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:37:36.571 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:37:36.592+0000 7f7fc8f06f00 -1 osd.2 0 log_to_monitors {default=true} 2022-03-01T06:37:36.899 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:37:36.900 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:36.900 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:37:36.901 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:37:36.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:36.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:37:37.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:37:38.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:37:38.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:37:38.245 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:37:38.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-03-01T06:37:38.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:37:38.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:37:38.577 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/49244,v1:127.0.0.1:6819/49244] [v2:127.0.0.1:6820/49244,v1:127.0.0.1:6821/49244] exists,up 8819d842-0ff0-4da3-9a3e-6344e0c5f1d5 2022-03-01T06:37:38.578 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:37:38.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:37:38.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:37:38.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2022-03-01T06:37:39.234 INFO:tasks.workunit.client.0.smithi169.stderr:noout is set 2022-03-01T06:37:39.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2022-03-01T06:37:39.864 INFO:tasks.workunit.client.0.smithi169.stderr:noin is set 2022-03-01T06:37:39.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2022-03-01T06:37:40.463 INFO:tasks.workunit.client.0.smithi169.stderr:nodown is set 2022-03-01T06:37:40.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2022-03-01T06:37:40.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2022-03-01T06:37:41.244 INFO:tasks.workunit.client.0.smithi169.stderr:pool 'test' already exists 2022-03-01T06:37:41.253 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2022-03-01T06:37:42.254 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2022-03-01T06:37:42.896 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 size to 3 2022-03-01T06:37:42.905 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2022-03-01T06:37:43.549 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 min_size to 2 2022-03-01T06:37:43.556 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2022-03-01T06:37:44.208 INFO:tasks.workunit.client.0.smithi169.stderr:set pool 1 pg_autoscale_mode to on 2022-03-01T06:37:44.216 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2022-03-01T06:37:44.216 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:37:44.217 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:37:44.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:37:44.542 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:37:44.542 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:37:44.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:37:44.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:37:44.543 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:37:44.671 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836486 2022-03-01T06:37:44.672 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836486 2022-03-01T06:37:44.672 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486' 2022-03-01T06:37:44.673 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:37:44.673 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:37:44.794 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2022-03-01T06:37:44.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2022-03-01T06:37:44.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2022-03-01T06:37:44.795 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:37:44.795 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:37:44.918 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2022-03-01T06:37:44.919 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2022-03-01T06:37:44.919 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2022-03-01T06:37:44.919 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:37:44.919 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2022-03-01T06:37:44.920 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:37:44.921 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:37:44.921 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2022-03-01T06:37:44.921 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:37:44.921 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836486 2022-03-01T06:37:44.922 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2022-03-01T06:37:44.922 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836486 2022-03-01T06:37:44.923 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:37:45.247 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836484 -lt 21474836486 2022-03-01T06:37:45.248 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:37:46.249 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:37:46.250 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:37:46.585 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836486 2022-03-01T06:37:46.585 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:37:46.586 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2022-03-01T06:37:46.586 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:37:46.587 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:37:46.587 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2022-03-01T06:37:46.587 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:37:46.588 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2022-03-01T06:37:46.588 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2022-03-01T06:37:46.589 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 42949672965 2022-03-01T06:37:46.589 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:37:46.911 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2022-03-01T06:37:46.911 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:37:46.912 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2022-03-01T06:37:46.913 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:37:46.913 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:37:46.914 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2022-03-01T06:37:46.914 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:37:46.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2022-03-01T06:37:46.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2022-03-01T06:37:46.915 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 64424509444 2022-03-01T06:37:46.915 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:37:47.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2022-03-01T06:37:47.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2022-03-01T06:37:47.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:37:47.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:37:47.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:37:47.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:37:47.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2022-03-01T06:37:47.242 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:37:47.242 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:37:47.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:37:47.243 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:37:47.243 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:37:47.331 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:37:47.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:37:47.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:37:47.332 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:37:47.333 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:37:47.654 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:37:47.654 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:37:47.655 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:37:47.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:37:47.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:37:47.656 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:37:47.779 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2022-03-01T06:37:47.779 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2022-03-01T06:37:47.780 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2022-03-01T06:37:47.780 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:37:47.780 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:37:47.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2022-03-01T06:37:47.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2022-03-01T06:37:47.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2022-03-01T06:37:47.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:37:47.904 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:37:48.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509447 2022-03-01T06:37:48.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509447 2022-03-01T06:37:48.030 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2022-03-01T06:37:48.030 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:37:48.030 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2022-03-01T06:37:48.031 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:37:48.031 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:37:48.031 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2022-03-01T06:37:48.031 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:37:48.032 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2022-03-01T06:37:48.032 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2022-03-01T06:37:48.033 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836489 2022-03-01T06:37:48.033 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:37:48.358 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2022-03-01T06:37:48.359 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:37:49.358 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:37:49.359 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:37:49.730 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836489 2022-03-01T06:37:49.731 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:37:49.731 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2022-03-01T06:37:49.732 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:37:49.734 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:37:49.734 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2022-03-01T06:37:49.735 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:37:49.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2022-03-01T06:37:49.735 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2022-03-01T06:37:49.736 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 42949672968 2022-03-01T06:37:49.736 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:37:50.056 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2022-03-01T06:37:50.057 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:37:50.057 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509447 2022-03-01T06:37:50.058 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:37:50.060 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:37:50.061 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2022-03-01T06:37:50.061 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:37:50.061 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509447 2022-03-01T06:37:50.062 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2022-03-01T06:37:50.064 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 64424509447 2022-03-01T06:37:50.065 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:37:50.380 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509447 2022-03-01T06:37:50.381 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:37:50.381 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:37:50.381 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:37:50.805 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:37:50.806 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:37:50.806 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:37:50.806 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:37:50.807 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:37:50.807 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:37:50.808 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:37:50.808 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:37:51.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-03-01T06:37:51.119 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:37:51.120 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:37:51.120 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:37:51.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-03-01T06:37:51.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-03-01T06:37:51.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-03-01T06:37:51.543 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2022-03-01T06:37:51.543 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2022-03-01T06:37:51.847 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:37:51.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2022-03-01T06:37:51.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2022-03-01T06:37:51.856 INFO:tasks.workunit.client.0.smithi169.stdout:primary and soon to be divergent is 1 2022-03-01T06:37:51.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2022-03-01T06:37:52.158 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:37:52.159 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-03-01T06:37:41.662216+0000 0'0 25:11 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:37:40.865883+0000 0'0 2022-03-01T06:37:40.865883+0000 0 2022-03-01T06:37:52.159 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:37:52.159 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:37:52.160 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:37:52.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2022-03-01T06:37:52.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2022-03-01T06:37:52.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2022-03-01T06:37:52.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2022-03-01T06:37:52.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2022-03-01T06:37:52.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2022-03-01T06:37:52.171 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2022-03-01T06:37:52.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2022-03-01T06:37:52.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2022-03-01T06:37:52.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2022-03-01T06:37:52.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2022-03-01T06:37:52.173 INFO:tasks.workunit.client.0.smithi169.stdout:writing initial objects 2022-03-01T06:37:52.173 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2022-03-01T06:37:52.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2022-03-01T06:37:52.272 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.272 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2022-03-01T06:37:52.364 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.365 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2022-03-01T06:37:52.465 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.465 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2022-03-01T06:37:52.592 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.592 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2022-03-01T06:37:52.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2022-03-01T06:37:52.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2022-03-01T06:37:52.901 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2022-03-01T06:37:52.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:52.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2022-03-01T06:37:53.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.086 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2022-03-01T06:37:53.195 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.196 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2022-03-01T06:37:53.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2022-03-01T06:37:53.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.472 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2022-03-01T06:37:53.572 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.572 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2022-03-01T06:37:53.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.665 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2022-03-01T06:37:53.756 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.757 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2022-03-01T06:37:53.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2022-03-01T06:37:53.969 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:53.970 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2022-03-01T06:37:54.084 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.084 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2022-03-01T06:37:54.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2022-03-01T06:37:54.268 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.269 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2022-03-01T06:37:54.377 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.378 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2022-03-01T06:37:54.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2022-03-01T06:37:54.645 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.646 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2022-03-01T06:37:54.754 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.755 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2022-03-01T06:37:54.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2022-03-01T06:37:54.972 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:54.973 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2022-03-01T06:37:55.065 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.065 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2022-03-01T06:37:55.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2022-03-01T06:37:55.300 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.300 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2022-03-01T06:37:55.392 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.392 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2022-03-01T06:37:55.484 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.485 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2022-03-01T06:37:55.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.577 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2022-03-01T06:37:55.669 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.675 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2022-03-01T06:37:55.803 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.804 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2022-03-01T06:37:55.899 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.900 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2022-03-01T06:37:55.996 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:55.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2022-03-01T06:37:56.096 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.097 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2022-03-01T06:37:56.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2022-03-01T06:37:56.290 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.291 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2022-03-01T06:37:56.407 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.408 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2022-03-01T06:37:56.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.533 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2022-03-01T06:37:56.625 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.625 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2022-03-01T06:37:56.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2022-03-01T06:37:56.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.810 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2022-03-01T06:37:56.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:56.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2022-03-01T06:37:57.027 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2022-03-01T06:37:57.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2022-03-01T06:37:57.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.246 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2022-03-01T06:37:57.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.380 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2022-03-01T06:37:57.480 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.481 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2022-03-01T06:37:57.589 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.589 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2022-03-01T06:37:57.731 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.732 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2022-03-01T06:37:57.824 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.825 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2022-03-01T06:37:57.916 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:57.917 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2022-03-01T06:37:58.017 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.017 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2022-03-01T06:37:58.109 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.110 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2022-03-01T06:37:58.252 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.252 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2022-03-01T06:37:58.352 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.353 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2022-03-01T06:37:58.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2022-03-01T06:37:58.587 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.588 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2022-03-01T06:37:58.688 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.688 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2022-03-01T06:37:58.780 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.781 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2022-03-01T06:37:58.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2022-03-01T06:37:58.990 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:58.991 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2022-03-01T06:37:59.091 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.092 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2022-03-01T06:37:59.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2022-03-01T06:37:59.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.309 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2022-03-01T06:37:59.434 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.435 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2022-03-01T06:37:59.526 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2022-03-01T06:37:59.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2022-03-01T06:37:59.711 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.711 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2022-03-01T06:37:59.803 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.804 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2022-03-01T06:37:59.937 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:37:59.938 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2022-03-01T06:38:00.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.080 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2022-03-01T06:38:00.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2022-03-01T06:38:00.264 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.265 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2022-03-01T06:38:00.381 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.382 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2022-03-01T06:38:00.483 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.483 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2022-03-01T06:38:00.608 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.609 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2022-03-01T06:38:00.751 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.751 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2022-03-01T06:38:00.843 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.843 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2022-03-01T06:38:00.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:00.935 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2022-03-01T06:38:01.036 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.036 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2022-03-01T06:38:01.128 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.129 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2022-03-01T06:38:01.254 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.255 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2022-03-01T06:38:01.397 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.397 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2022-03-01T06:38:01.488 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.489 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2022-03-01T06:38:01.581 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.581 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2022-03-01T06:38:01.673 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.674 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2022-03-01T06:38:01.782 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.783 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2022-03-01T06:38:01.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:01.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2022-03-01T06:38:02.000 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.001 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2022-03-01T06:38:02.093 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.093 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2022-03-01T06:38:02.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2022-03-01T06:38:02.277 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.278 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2022-03-01T06:38:02.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.371 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2022-03-01T06:38:02.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.471 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2022-03-01T06:38:02.588 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.588 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2022-03-01T06:38:02.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2022-03-01T06:38:02.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2022-03-01T06:38:02.798 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:38:02.798 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2022-03-01T06:38:02.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:38:02.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:38:02.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:38:02.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:38:02.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:38:02.800 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:38:02.800 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:38:02.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:38:02.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:38:02.801 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:38:02.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:38:02.854 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:38:02.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:38:02.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:38:02.855 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:38:03.177 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:38:03.178 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:38:03.178 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:38:03.178 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:38:03.179 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:38:03.179 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:38:03.318 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836494 2022-03-01T06:38:03.319 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836494 2022-03-01T06:38:03.319 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494' 2022-03-01T06:38:03.319 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:38:03.319 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:38:03.444 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672973 2022-03-01T06:38:03.444 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672973 2022-03-01T06:38:03.444 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672973' 2022-03-01T06:38:03.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:38:03.445 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:38:03.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2022-03-01T06:38:03.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2022-03-01T06:38:03.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672973 2-64424509452' 2022-03-01T06:38:03.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:38:03.569 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836494 2022-03-01T06:38:03.569 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:38:03.570 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:38:03.570 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836494 2022-03-01T06:38:03.571 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:38:03.571 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836494 2022-03-01T06:38:03.571 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2022-03-01T06:38:03.572 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 21474836494 2022-03-01T06:38:03.572 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:03.890 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836494 2022-03-01T06:38:03.890 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:04.891 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:38:04.893 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:05.257 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836494 2022-03-01T06:38:05.257 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:38:05.258 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672973 2022-03-01T06:38:05.258 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:38:05.258 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2022-03-01T06:38:05.259 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2022-03-01T06:38:05.259 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:38:05.260 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672973 2022-03-01T06:38:05.260 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2022-03-01T06:38:05.261 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.1 seq 42949672973 2022-03-01T06:38:05.261 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2022-03-01T06:38:05.581 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672973 -lt 42949672973 2022-03-01T06:38:05.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:38:05.582 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2022-03-01T06:38:05.582 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:38:05.583 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2022-03-01T06:38:05.583 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2022-03-01T06:38:05.584 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:38:05.584 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2022-03-01T06:38:05.585 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2022-03-01T06:38:05.585 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.2 seq 64424509452 2022-03-01T06:38:05.586 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2022-03-01T06:38:05.906 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2022-03-01T06:38:05.906 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T06:38:05.907 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:38:05.907 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:38:06.325 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2022-03-01T06:38:06.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T06:38:06.326 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T06:38:06.326 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T06:38:06.327 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T06:38:06.327 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T06:38:06.327 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T06:38:06.327 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T06:38:06.665 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2022-03-01T06:38:06.666 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2022-03-01T06:38:06.666 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T06:38:06.666 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T06:38:07.091 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2022-03-01T06:38:07.092 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2022-03-01T06:38:07.092 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2022-03-01T06:38:07.092 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2022-03-01T06:38:07.092 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2022-03-01T06:38:07.093 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2022-03-01T06:38:07.093 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2022-03-01T06:38:07.093 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2022-03-01T06:38:07.419 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2022-03-01T06:38:07.420 INFO:tasks.workunit.client.0.smithi169.stdout:blackholing osds 0 2 2022-03-01T06:38:07.421 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2022-03-01T06:38:07.422 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2022-03-01T06:38:07.720 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:38:07.720 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2022-03-01T06:37:41.662216+0000 31'100 31:111 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:37:40.865883+0000 0'0 2022-03-01T06:37:40.865883+0000 0 2022-03-01T06:38:07.721 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:38:07.721 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:38:07.722 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:38:07.737 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:07.737 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2022-03-01T06:38:07.738 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2022-03-01T06:38:07.738 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2022-03-01T06:38:07.739 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:38:07.740 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:38:07.741 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:38:07.741 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.0.asok 2022-03-01T06:38:07.742 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2022-03-01T06:38:07.742 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.0.asok config set objectstore_blackhole 1 2022-03-01T06:38:07.812 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:38:07.813 INFO:tasks.workunit.client.0.smithi169.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-03-01T06:38:07.813 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:38:07.819 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:07.821 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2022-03-01T06:38:07.821 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2022-03-01T06:38:07.822 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2022-03-01T06:38:07.822 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2022-03-01T06:38:07.822 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:38:07.822 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:38:07.823 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31520/ceph-osd.2.asok 2022-03-01T06:38:07.823 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2022-03-01T06:38:07.823 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.31520/ceph-osd.2.asok config set objectstore_blackhole 1 2022-03-01T06:38:07.902 INFO:tasks.workunit.client.0.smithi169.stdout:{ 2022-03-01T06:38:07.902 INFO:tasks.workunit.client.0.smithi169.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2022-03-01T06:38:07.902 INFO:tasks.workunit.client.0.smithi169.stdout:} 2022-03-01T06:38:07.909 INFO:tasks.workunit.client.0.smithi169.stdout:create new divergent objects 2022-03-01T06:38:07.910 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2022-03-01T06:38:07.910 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2022-03-01T06:38:07.911 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2022-03-01T06:38:07.911 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2022-03-01T06:38:07.911 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2022-03-01T06:38:07.912 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2022-03-01T06:38:07.912 INFO:tasks.workunit.client.0.smithi169.stdout:writing divergent objects 2022-03-01T06:38:07.913 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2022-03-01T06:38:07.913 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2022-03-01T06:38:07.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2022-03-01T06:38:07.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2022-03-01T06:38:07.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2022-03-01T06:38:07.914 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2022-03-01T06:38:07.915 INFO:tasks.workunit.client.0.smithi169.stdout:remove divergent objects 2022-03-01T06:38:07.916 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2022-03-01T06:38:07.916 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2022-03-01T06:38:07.917 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-03-01T06:38:07.917 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2022-03-01T06:38:07.917 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2022-03-01T06:38:07.917 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2022-03-01T06:38:07.918 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2022-03-01T06:38:07.918 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2022-03-01T06:38:07.918 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2022-03-01T06:38:07.918 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2022-03-01T06:38:07.919 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2022-03-01T06:38:17.918 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2022-03-01T06:38:17.919 INFO:tasks.workunit.client.0.smithi169.stderr:rados: no process found 2022-03-01T06:38:17.920 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2022-03-01T06:38:17.920 INFO:tasks.workunit.client.0.smithi169.stdout:killing all the osds 2022-03-01T06:38:17.921 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2022-03-01T06:38:18.232 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:38:18.232 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2022-03-01T06:37:41.662216+0000 31'106 31:117 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:37:40.865883+0000 0'0 2022-03-01T06:37:40.865883+0000 0 2022-03-01T06:38:18.232 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:38:18.233 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:38:18.233 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:38:18.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2022-03-01T06:38:18.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T06:38:18.243 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T06:38:18.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T06:38:18.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T06:38:18.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T06:38:18.349 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T06:38:18.349 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2022-03-01T06:38:18.349 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2022-03-01T06:38:18.804 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already down. 2022-03-01T06:38:18.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2022-03-01T06:38:18.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2022-03-01T06:38:19.271 INFO:tasks.workunit.client.0.smithi169.stderr:osd.1 is already down. 2022-03-01T06:38:19.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2022-03-01T06:38:19.280 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2022-03-01T06:38:19.721 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already down. 2022-03-01T06:38:19.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:19.729 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2022-03-01T06:38:20.170 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already out. 2022-03-01T06:38:20.178 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:20.179 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2022-03-01T06:38:20.789 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already out. 2022-03-01T06:38:20.797 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2022-03-01T06:38:20.798 INFO:tasks.workunit.client.0.smithi169.stdout:bringing up non_divergent 0 2 2022-03-01T06:38:20.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2022-03-01T06:38:21.102 INFO:tasks.workunit.client.0.smithi169.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2022-03-01T06:38:21.102 INFO:tasks.workunit.client.0.smithi169.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2022-03-01T06:37:41.662216+0000 31'106 31:117 [1,0,2] 1 [1,0,2] 1 0'0 2022-03-01T06:37:40.865883+0000 0'0 2022-03-01T06:37:40.865883+0000 0 2022-03-01T06:38:21.103 INFO:tasks.workunit.client.0.smithi169.stdout: 2022-03-01T06:38:21.103 INFO:tasks.workunit.client.0.smithi169.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2022-03-01T06:38:21.103 INFO:tasks.workunit.client.0.smithi169.stderr:dumped pgs 2022-03-01T06:38:21.116 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:21.116 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2022-03-01T06:38:21.116 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:38:21.117 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:38:21.117 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2022-03-01T06:38:21.117 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:38:21.118 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2022-03-01T06:38:21.118 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:38:21.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:38:21.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:38:21.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:38:21.119 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2022-03-01T06:38:21.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2022-03-01T06:38:21.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:38:21.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:38:21.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:38:21.122 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:38:21.123 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:38:21.123 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:38:21.123 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:38:21.123 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:38:21.124 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:38:21.124 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:38:21.124 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:38:21.124 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:38:21.125 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:38:21.125 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:38:21.125 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:38:21.125 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:38:21.125 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:38:21.126 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:38:21.126 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:38:21.126 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2022-03-01T06:38:21.126 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2022-03-01T06:38:21.127 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.0 2022-03-01T06:38:21.128 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:38:21.128 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2022-03-01T06:38:21.129 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2022-03-01T06:38:21.129 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:38:21.129 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:38:21.129 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:38:21.142 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:21.163+0000 7fe0f3baff00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:38:21.146 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:21.167+0000 7fe0f3baff00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:38:21.147 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:21.168+0000 7fe0f3baff00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:38:21.444 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2022-03-01T06:38:21.444 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:38:21.444 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2022-03-01T06:38:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:38:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:38:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:38:21.445 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:38:21.446 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:38:21.447 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:38:21.447 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:38:21.668 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:21.689+0000 7fe0f3baff00 -1 Falling back to public interface 2022-03-01T06:38:21.845 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:38:22.540 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:22.561+0000 7fe0f3baff00 -1 osd.0 31 log_to_monitors {default=true} 2022-03-01T06:38:22.771 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:38:22.771 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:38:22.772 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:38:22.772 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:38:22.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:38:22.773 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2022-03-01T06:38:23.135 INFO:tasks.workunit.client.0.smithi169.stdout:osd.0 up out weight 0 up_from 39 up_thru 0 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/53325,v1:127.0.0.1:6803/53325] [v2:127.0.0.1:6804/53325,v1:127.0.0.1:6805/53325] exists,up e1cd9c7c-2111-4f83-aa80-69209d7cbd2f 2022-03-01T06:38:23.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:38:23.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:38:23.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:38:23.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:23.142 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2022-03-01T06:38:23.143 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2022-03-01T06:38:23.143 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2022-03-01T06:38:23.143 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2022-03-01T06:38:23.143 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2022-03-01T06:38:23.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2022-03-01T06:38:23.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2022-03-01T06:38:23.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2022-03-01T06:38:23.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2022-03-01T06:38:23.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2022-03-01T06:38:23.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2022-03-01T06:38:23.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2022-03-01T06:38:23.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2022-03-01T06:38:23.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2022-03-01T06:38:23.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2022-03-01T06:38:23.146 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2022-03-01T06:38:23.146 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2022-03-01T06:38:23.147 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2022-03-01T06:38:23.147 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2022-03-01T06:38:23.147 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T06:38:23.147 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T06:38:23.147 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:38:23.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' 2022-03-01T06:38:23.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2022-03-01T06:38:23.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2022-03-01T06:38:23.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2022-03-01T06:38:23.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2022-03-01T06:38:23.150 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2022-03-01T06:38:23.150 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:38:23.150 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2022-03-01T06:38:23.150 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2022-03-01T06:38:23.151 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2022-03-01T06:38:23.151 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2022-03-01T06:38:23.151 INFO:tasks.workunit.client.0.smithi169.stdout:start osd.2 2022-03-01T06:38:23.152 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=03da9435-ccbd-45e1-be8b-88e9f3ce57a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.31520/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2022-03-01T06:38:23.153 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2022-03-01T06:38:23.153 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2022-03-01T06:38:23.153 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2022-03-01T06:38:23.153 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2022-03-01T06:38:23.154 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2022-03-01T06:38:23.165 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:23.187+0000 7f620a9d3f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:38:23.167 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:23.189+0000 7f620a9d3f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:38:23.168 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:23.189+0000 7f620a9d3f00 -1 WARNING: all dangerous and experimental features are enabled. 2022-03-01T06:38:23.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2022-03-01T06:38:23.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2022-03-01T06:38:23.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2022-03-01T06:38:23.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2022-03-01T06:38:23.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2022-03-01T06:38:23.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:38:23.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2022-03-01T06:38:23.480 INFO:tasks.workunit.client.0.smithi169.stdout:0 2022-03-01T06:38:23.480 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:38:23.481 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:38:23.684 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:23.705+0000 7f620a9d3f00 -1 Falling back to public interface 2022-03-01T06:38:23.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:38:24.786 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T06:38:24.808+0000 7f620a9d3f00 -1 osd.2 31 log_to_monitors {default=true} 2022-03-01T06:38:24.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:38:24.809 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:38:24.810 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2022-03-01T06:38:24.810 INFO:tasks.workunit.client.0.smithi169.stdout:1 2022-03-01T06:38:24.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:38:24.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:38:25.292 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2022-03-01T06:38:26.294 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2022-03-01T06:38:26.295 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2022-03-01T06:38:26.295 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2022-03-01T06:38:26.296 INFO:tasks.workunit.client.0.smithi169.stdout:2 2022-03-01T06:38:26.297 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2022-03-01T06:38:26.297 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2022-03-01T06:38:27.184 INFO:tasks.workunit.client.0.smithi169.stdout:osd.2 up out weight 0 up_from 41 up_thru 0 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6810/53519,v1:127.0.0.1:6811/53519] [v2:127.0.0.1:6812/53519,v1:127.0.0.1:6813/53519] exists,up 8819d842-0ff0-4da3-9a3e-6344e0c5f1d5 2022-03-01T06:38:27.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2022-03-01T06:38:27.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2022-03-01T06:38:27.186 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2022-03-01T06:38:27.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:27.187 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2022-03-01T06:38:28.093 INFO:tasks.workunit.client.0.smithi169.stderr:osd.0 is already in. 2022-03-01T06:38:28.101 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2022-03-01T06:38:28.101 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2022-03-01T06:38:28.788 INFO:tasks.workunit.client.0.smithi169.stderr:osd.2 is already in. 2022-03-01T06:38:28.797 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T06:38:28.798 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2022-03-01T06:38:28.798 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T06:38:28.798 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T06:38:28.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T06:38:28.799 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T06:38:28.799 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T06:38:28.799 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T06:38:28.799 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T06:38:28.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T06:38:28.800 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T06:38:28.801 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T06:38:28.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T06:38:28.855 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T06:38:28.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T06:38:28.856 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T06:38:28.856 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T06:38:29.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2022-03-01T06:38:29.183 INFO:tasks.workunit.client.0.smithi169.stderr:1 2022-03-01T06:38:29.183 INFO:tasks.workunit.client.0.smithi169.stderr:2' 2022-03-01T06:38:29.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T06:38:29.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:38:29.184 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2022-03-01T06:38:29.307 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724547 2022-03-01T06:38:29.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724547 2022-03-01T06:38:29.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724547' 2022-03-01T06:38:29.308 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:38:29.308 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2022-03-01T06:38:29.396 INFO:tasks.workunit.client.0.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.1 2022-03-01T06:38:29.397 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2022-03-01T06:38:29.398 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2022-03-01T06:38:29.398 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2022-03-01T06:38:29.398 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2022-03-01T06:38:29.398 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2022-03-01T06:38:29.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659138 2022-03-01T06:38:29.521 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659138 2022-03-01T06:38:29.522 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724547 2-176093659138' 2022-03-01T06:38:29.522 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2022-03-01T06:38:29.522 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-167503724547 2022-03-01T06:38:29.522 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2022-03-01T06:38:29.523 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2022-03-01T06:38:29.524 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724547 2022-03-01T06:38:29.524 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2022-03-01T06:38:29.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724547 2022-03-01T06:38:29.524 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2022-03-01T06:38:29.525 INFO:tasks.workunit.client.0.smithi169.stdout:waiting osd.0 seq 167503724547 2022-03-01T06:38:29.525 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:29.844 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:29.845 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:30.845 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2022-03-01T06:38:30.846 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:31.183 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:31.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:32.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 299 -eq 0 ']' 2022-03-01T06:38:32.185 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:32.518 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:32.518 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:33.519 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 298 -eq 0 ']' 2022-03-01T06:38:33.520 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:33.850 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:33.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:34.851 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 297 -eq 0 ']' 2022-03-01T06:38:34.852 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:35.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:35.184 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:36.185 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 296 -eq 0 ']' 2022-03-01T06:38:36.185 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:36.518 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:36.519 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:37.523 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 295 -eq 0 ']' 2022-03-01T06:38:37.523 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:37.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:37.852 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:38.853 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 294 -eq 0 ']' 2022-03-01T06:38:38.853 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:39.188 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:39.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:40.189 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 293 -eq 0 ']' 2022-03-01T06:38:40.190 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:40.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:40.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:41.525 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 292 -eq 0 ']' 2022-03-01T06:38:41.526 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:41.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:41.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:42.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 291 -eq 0 ']' 2022-03-01T06:38:42.862 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:43.196 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:43.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:44.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 290 -eq 0 ']' 2022-03-01T06:38:44.198 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:44.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:44.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:45.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 289 -eq 0 ']' 2022-03-01T06:38:45.531 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:45.861 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:45.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:46.862 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 288 -eq 0 ']' 2022-03-01T06:38:46.863 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:47.194 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:47.194 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:48.195 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 287 -eq 0 ']' 2022-03-01T06:38:48.195 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:48.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:48.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:49.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 286 -eq 0 ']' 2022-03-01T06:38:49.546 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:49.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:49.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:50.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 285 -eq 0 ']' 2022-03-01T06:38:50.878 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:51.218 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:51.218 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:52.219 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 284 -eq 0 ']' 2022-03-01T06:38:52.220 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:52.552 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:52.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:53.553 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 283 -eq 0 ']' 2022-03-01T06:38:53.554 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:53.907 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:53.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:54.908 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 282 -eq 0 ']' 2022-03-01T06:38:54.909 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:55.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:55.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:56.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 281 -eq 0 ']' 2022-03-01T06:38:56.241 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:56.574 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:56.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:57.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 280 -eq 0 ']' 2022-03-01T06:38:57.576 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:57.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:57.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:38:58.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 279 -eq 0 ']' 2022-03-01T06:38:58.905 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:38:59.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:38:59.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:00.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 278 -eq 0 ']' 2022-03-01T06:39:00.240 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:00.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:00.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:01.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 277 -eq 0 ']' 2022-03-01T06:39:01.576 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:01.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:01.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:02.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 276 -eq 0 ']' 2022-03-01T06:39:02.904 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:03.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:03.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:04.231 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 275 -eq 0 ']' 2022-03-01T06:39:04.232 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:04.564 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:04.565 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:05.565 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 274 -eq 0 ']' 2022-03-01T06:39:05.566 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:05.897 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:05.898 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:06.897 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 273 -eq 0 ']' 2022-03-01T06:39:06.898 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:07.230 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:07.231 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:08.227 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 272 -eq 0 ']' 2022-03-01T06:39:08.228 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:08.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:08.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:09.559 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 271 -eq 0 ']' 2022-03-01T06:39:09.560 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:09.885 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:09.885 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:10.886 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 270 -eq 0 ']' 2022-03-01T06:39:10.886 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:11.221 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:11.221 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:12.222 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 269 -eq 0 ']' 2022-03-01T06:39:12.223 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:12.554 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:12.555 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:13.555 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 268 -eq 0 ']' 2022-03-01T06:39:13.556 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:13.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:13.894 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:14.894 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 267 -eq 0 ']' 2022-03-01T06:39:14.895 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:15.228 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:15.228 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:16.229 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 266 -eq 0 ']' 2022-03-01T06:39:16.230 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:16.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:16.560 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:17.561 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 265 -eq 0 ']' 2022-03-01T06:39:17.561 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:17.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:17.893 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:18.894 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 264 -eq 0 ']' 2022-03-01T06:39:18.895 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:19.226 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:19.227 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:20.227 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 263 -eq 0 ']' 2022-03-01T06:39:20.228 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:20.557 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:20.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:21.558 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 262 -eq 0 ']' 2022-03-01T06:39:21.559 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:21.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:21.915 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:22.916 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 261 -eq 0 ']' 2022-03-01T06:39:22.916 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:23.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:23.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:24.244 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 260 -eq 0 ']' 2022-03-01T06:39:24.245 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:24.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:24.582 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:25.585 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 259 -eq 0 ']' 2022-03-01T06:39:25.585 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:25.909 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:25.909 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:26.910 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 258 -eq 0 ']' 2022-03-01T06:39:26.911 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:27.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:27.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:28.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 257 -eq 0 ']' 2022-03-01T06:39:28.241 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:28.573 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:28.575 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:29.574 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 256 -eq 0 ']' 2022-03-01T06:39:29.575 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:29.905 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:29.906 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:30.907 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 255 -eq 0 ']' 2022-03-01T06:39:30.907 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:31.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:31.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:32.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 254 -eq 0 ']' 2022-03-01T06:39:32.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:32.569 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:32.569 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:33.570 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 253 -eq 0 ']' 2022-03-01T06:39:33.571 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:33.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:33.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:34.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 252 -eq 0 ']' 2022-03-01T06:39:34.905 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:35.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:35.238 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:36.239 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 251 -eq 0 ']' 2022-03-01T06:39:36.239 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:36.571 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:36.571 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:37.572 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 250 -eq 0 ']' 2022-03-01T06:39:37.572 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:37.901 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:37.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:38.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 249 -eq 0 ']' 2022-03-01T06:39:38.903 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:39.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:39.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:40.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 248 -eq 0 ']' 2022-03-01T06:39:40.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:40.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:40.569 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:41.570 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 247 -eq 0 ']' 2022-03-01T06:39:41.570 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:41.909 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:41.910 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:42.911 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 246 -eq 0 ']' 2022-03-01T06:39:42.911 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:43.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:43.242 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:44.243 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 245 -eq 0 ']' 2022-03-01T06:39:44.244 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:44.578 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:44.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:45.579 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 244 -eq 0 ']' 2022-03-01T06:39:45.580 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:45.912 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:45.912 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:46.913 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 243 -eq 0 ']' 2022-03-01T06:39:46.914 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:47.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:47.240 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:48.241 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 242 -eq 0 ']' 2022-03-01T06:39:48.242 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:48.568 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:48.569 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:49.569 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 241 -eq 0 ']' 2022-03-01T06:39:49.570 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:49.902 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:49.903 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:50.904 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 240 -eq 0 ']' 2022-03-01T06:39:50.904 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:51.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:51.236 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:52.237 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 239 -eq 0 ']' 2022-03-01T06:39:52.238 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:52.613 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:52.613 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:53.614 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 238 -eq 0 ']' 2022-03-01T06:39:53.614 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:53.949 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:53.950 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:54.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 237 -eq 0 ']' 2022-03-01T06:39:54.951 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:55.282 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:55.282 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:56.283 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 236 -eq 0 ']' 2022-03-01T06:39:56.284 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:56.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:56.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:57.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 235 -eq 0 ']' 2022-03-01T06:39:57.620 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:57.954 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:57.954 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:39:58.955 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 234 -eq 0 ']' 2022-03-01T06:39:58.955 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:39:59.285 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:39:59.285 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:00.286 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 233 -eq 0 ']' 2022-03-01T06:40:00.287 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:00.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:00.618 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:01.619 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 232 -eq 0 ']' 2022-03-01T06:40:01.619 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:01.951 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:01.952 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:02.953 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 231 -eq 0 ']' 2022-03-01T06:40:02.953 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:03.286 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:03.287 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:04.287 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 230 -eq 0 ']' 2022-03-01T06:40:04.288 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:04.622 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:04.622 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:05.623 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 229 -eq 0 ']' 2022-03-01T06:40:05.624 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:05.957 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:05.958 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:06.958 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 228 -eq 0 ']' 2022-03-01T06:40:06.959 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:07.314 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:07.314 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:08.315 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 227 -eq 0 ']' 2022-03-01T06:40:08.315 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:08.647 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:08.647 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:09.648 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 226 -eq 0 ']' 2022-03-01T06:40:09.648 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:09.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:09.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:10.988 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 225 -eq 0 ']' 2022-03-01T06:40:10.989 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:11.322 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:11.323 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:12.323 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 224 -eq 0 ']' 2022-03-01T06:40:12.324 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:12.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:12.656 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:13.656 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 223 -eq 0 ']' 2022-03-01T06:40:13.657 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:14.006 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:14.007 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:15.007 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 222 -eq 0 ']' 2022-03-01T06:40:15.008 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:15.337 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:15.338 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:16.339 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 221 -eq 0 ']' 2022-03-01T06:40:16.339 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:16.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:16.664 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:17.665 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 220 -eq 0 ']' 2022-03-01T06:40:17.666 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:17.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:17.997 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:18.999 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 219 -eq 0 ']' 2022-03-01T06:40:18.999 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:19.326 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:19.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:20.327 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 218 -eq 0 ']' 2022-03-01T06:40:20.328 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:20.661 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:20.661 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:21.662 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 217 -eq 0 ']' 2022-03-01T06:40:21.663 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:21.992 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:21.993 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:22.994 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 216 -eq 0 ']' 2022-03-01T06:40:22.994 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:23.378 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:23.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:24.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 215 -eq 0 ']' 2022-03-01T06:40:24.380 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:24.711 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:24.712 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:25.712 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 214 -eq 0 ']' 2022-03-01T06:40:25.713 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:26.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:26.046 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:27.047 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 213 -eq 0 ']' 2022-03-01T06:40:27.047 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:27.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:27.379 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:28.380 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 212 -eq 0 ']' 2022-03-01T06:40:28.381 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:28.714 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:28.714 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:29.715 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 211 -eq 0 ']' 2022-03-01T06:40:29.715 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:30.048 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:30.048 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:31.049 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 210 -eq 0 ']' 2022-03-01T06:40:31.049 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:31.383 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:31.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:32.384 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 209 -eq 0 ']' 2022-03-01T06:40:32.387 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:32.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:32.717 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:33.718 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 208 -eq 0 ']' 2022-03-01T06:40:33.719 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:34.051 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:34.051 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:35.050 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 207 -eq 0 ']' 2022-03-01T06:40:35.051 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:35.376 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:35.377 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:36.377 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 206 -eq 0 ']' 2022-03-01T06:40:36.378 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:36.707 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:36.707 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:37.709 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 205 -eq 0 ']' 2022-03-01T06:40:37.710 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:38.039 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:38.039 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:39.040 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 204 -eq 0 ']' 2022-03-01T06:40:39.040 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:39.374 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:39.374 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:40.375 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 203 -eq 0 ']' 2022-03-01T06:40:40.376 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:40.706 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:40.706 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:41.707 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 202 -eq 0 ']' 2022-03-01T06:40:41.708 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:42.037 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:42.037 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:43.038 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 201 -eq 0 ']' 2022-03-01T06:40:43.039 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:43.369 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:43.369 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:44.370 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 200 -eq 0 ']' 2022-03-01T06:40:44.371 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:44.699 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:44.699 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:45.700 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 199 -eq 0 ']' 2022-03-01T06:40:45.701 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:46.033 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:46.034 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:47.035 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 198 -eq 0 ']' 2022-03-01T06:40:47.035 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:47.361 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:47.361 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:48.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 197 -eq 0 ']' 2022-03-01T06:40:48.363 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:48.693 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:48.694 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:49.694 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 196 -eq 0 ']' 2022-03-01T06:40:49.695 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:50.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:50.028 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:51.029 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 195 -eq 0 ']' 2022-03-01T06:40:51.030 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:51.361 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:51.361 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:52.362 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 194 -eq 0 ']' 2022-03-01T06:40:52.363 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:52.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:52.697 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:53.698 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 193 -eq 0 ']' 2022-03-01T06:40:53.700 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:54.070 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:54.071 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:55.071 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 192 -eq 0 ']' 2022-03-01T06:40:55.072 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:55.451 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:55.452 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:56.452 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 191 -eq 0 ']' 2022-03-01T06:40:56.452 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:56.782 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:56.783 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:57.783 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 190 -eq 0 ']' 2022-03-01T06:40:57.784 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:58.120 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:58.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:40:59.121 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 189 -eq 0 ']' 2022-03-01T06:40:59.121 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:40:59.455 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:40:59.455 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:00.455 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 188 -eq 0 ']' 2022-03-01T06:41:00.456 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:00.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:00.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:01.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 187 -eq 0 ']' 2022-03-01T06:41:01.814 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:02.144 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:02.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:03.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 186 -eq 0 ']' 2022-03-01T06:41:03.146 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:03.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:03.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:04.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 185 -eq 0 ']' 2022-03-01T06:41:04.479 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:04.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:04.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:05.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 184 -eq 0 ']' 2022-03-01T06:41:05.813 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:06.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:06.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:07.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 183 -eq 0 ']' 2022-03-01T06:41:07.147 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:07.480 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:07.481 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:08.481 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 182 -eq 0 ']' 2022-03-01T06:41:08.482 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:08.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:08.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:09.813 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 181 -eq 0 ']' 2022-03-01T06:41:09.814 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:10.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:10.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:11.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 180 -eq 0 ']' 2022-03-01T06:41:11.148 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:11.476 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:12.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 179 -eq 0 ']' 2022-03-01T06:41:12.478 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:12.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:12.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:13.815 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 178 -eq 0 ']' 2022-03-01T06:41:13.816 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:14.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:14.141 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:15.142 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 177 -eq 0 ']' 2022-03-01T06:41:15.143 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:15.475 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:15.475 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:16.476 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 176 -eq 0 ']' 2022-03-01T06:41:16.476 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:16.807 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:16.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:17.808 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 175 -eq 0 ']' 2022-03-01T06:41:17.809 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:18.139 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:18.140 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:19.140 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 174 -eq 0 ']' 2022-03-01T06:41:19.141 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:19.477 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:19.477 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:20.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 173 -eq 0 ']' 2022-03-01T06:41:20.478 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:20.802 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:20.802 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:21.804 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 172 -eq 0 ']' 2022-03-01T06:41:21.805 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:22.133 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:22.133 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:23.134 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 171 -eq 0 ']' 2022-03-01T06:41:23.135 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:23.465 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:23.465 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:24.466 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 170 -eq 0 ']' 2022-03-01T06:41:24.467 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:24.816 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:24.816 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:25.816 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 169 -eq 0 ']' 2022-03-01T06:41:25.817 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:26.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:26.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:27.149 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 168 -eq 0 ']' 2022-03-01T06:41:27.149 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:27.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:27.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:28.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 167 -eq 0 ']' 2022-03-01T06:41:28.480 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:28.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:28.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:29.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 166 -eq 0 ']' 2022-03-01T06:41:29.813 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:30.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:30.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:31.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 165 -eq 0 ']' 2022-03-01T06:41:31.147 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:31.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:31.478 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:32.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 164 -eq 0 ']' 2022-03-01T06:41:32.480 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:32.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:32.811 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:33.812 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 163 -eq 0 ']' 2022-03-01T06:41:33.813 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:34.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:34.147 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:35.148 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 162 -eq 0 ']' 2022-03-01T06:41:35.148 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:35.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:35.479 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:36.480 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 161 -eq 0 ']' 2022-03-01T06:41:36.481 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:36.815 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:36.815 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:37.815 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 160 -eq 0 ']' 2022-03-01T06:41:37.816 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:38.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:38.146 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:39.145 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 159 -eq 0 ']' 2022-03-01T06:41:39.146 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:39.493 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:39.494 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:40.494 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 158 -eq 0 ']' 2022-03-01T06:41:40.495 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:40.829 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:40.830 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:41.830 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 157 -eq 0 ']' 2022-03-01T06:41:41.832 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:42.161 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:42.161 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:43.162 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 156 -eq 0 ']' 2022-03-01T06:41:43.162 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:43.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:43.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:44.499 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 155 -eq 0 ']' 2022-03-01T06:41:44.500 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:44.830 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:44.831 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:45.831 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 154 -eq 0 ']' 2022-03-01T06:41:45.832 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:46.164 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:46.165 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:47.165 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 153 -eq 0 ']' 2022-03-01T06:41:47.166 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:47.502 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:47.502 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:48.503 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 152 -eq 0 ']' 2022-03-01T06:41:48.504 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:48.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:48.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:49.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 151 -eq 0 ']' 2022-03-01T06:41:49.838 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:50.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:50.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:51.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 150 -eq 0 ']' 2022-03-01T06:41:51.170 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:51.503 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:51.503 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:52.504 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 149 -eq 0 ']' 2022-03-01T06:41:52.505 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:52.836 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:52.836 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:53.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 148 -eq 0 ']' 2022-03-01T06:41:53.838 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:54.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:54.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:55.170 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 147 -eq 0 ']' 2022-03-01T06:41:55.172 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:55.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:55.512 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:56.513 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 146 -eq 0 ']' 2022-03-01T06:41:56.513 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:56.846 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:56.847 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:57.846 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 145 -eq 0 ']' 2022-03-01T06:41:57.847 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:58.176 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:58.177 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:41:59.177 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 144 -eq 0 ']' 2022-03-01T06:41:59.178 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:41:59.510 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:41:59.510 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:00.511 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 143 -eq 0 ']' 2022-03-01T06:42:00.511 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:00.841 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:00.841 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:01.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 142 -eq 0 ']' 2022-03-01T06:42:01.843 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:02.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:02.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:03.173 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 141 -eq 0 ']' 2022-03-01T06:42:03.174 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:03.505 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:03.505 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:04.506 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 140 -eq 0 ']' 2022-03-01T06:42:04.507 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:04.841 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:04.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:05.842 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 139 -eq 0 ']' 2022-03-01T06:42:05.843 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:06.172 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:06.173 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:07.174 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 138 -eq 0 ']' 2022-03-01T06:42:07.174 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:07.505 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:07.506 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:08.506 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 137 -eq 0 ']' 2022-03-01T06:42:08.507 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:08.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:08.837 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:09.838 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 136 -eq 0 ']' 2022-03-01T06:42:09.838 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:10.167 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:10.168 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:11.169 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 135 -eq 0 ']' 2022-03-01T06:42:11.169 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:11.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:11.498 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:12.499 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 134 -eq 0 ']' 2022-03-01T06:42:12.500 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:12.830 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:12.830 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:13.829 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 133 -eq 0 ']' 2022-03-01T06:42:13.829 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:14.161 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:14.162 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:15.162 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 132 -eq 0 ']' 2022-03-01T06:42:15.163 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:15.494 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:15.495 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:16.495 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 131 -eq 0 ']' 2022-03-01T06:42:16.495 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:16.827 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:16.827 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:17.828 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 130 -eq 0 ']' 2022-03-01T06:42:17.829 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:18.165 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:18.166 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:19.166 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 129 -eq 0 ']' 2022-03-01T06:42:19.167 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:19.496 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:19.496 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:20.497 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 128 -eq 0 ']' 2022-03-01T06:42:20.498 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:20.827 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:20.827 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:21.828 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 127 -eq 0 ']' 2022-03-01T06:42:21.828 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:22.157 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:22.158 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:23.158 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 126 -eq 0 ']' 2022-03-01T06:42:23.159 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:23.490 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:23.490 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:24.491 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 125 -eq 0 ']' 2022-03-01T06:42:24.492 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:24.824 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:24.824 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:25.825 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 124 -eq 0 ']' 2022-03-01T06:42:25.825 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:26.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:26.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:27.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 123 -eq 0 ']' 2022-03-01T06:42:27.209 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:27.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:27.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:28.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 122 -eq 0 ']' 2022-03-01T06:42:28.543 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:28.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:28.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:29.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 121 -eq 0 ']' 2022-03-01T06:42:29.879 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:30.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:30.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:31.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 120 -eq 0 ']' 2022-03-01T06:42:31.213 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:31.544 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:31.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:32.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 119 -eq 0 ']' 2022-03-01T06:42:32.546 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:32.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:32.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:33.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 118 -eq 0 ']' 2022-03-01T06:42:33.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:34.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:34.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:35.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 117 -eq 0 ']' 2022-03-01T06:42:35.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:35.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:35.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:36.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 116 -eq 0 ']' 2022-03-01T06:42:36.538 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:36.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:36.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:37.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 115 -eq 0 ']' 2022-03-01T06:42:37.871 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:38.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:38.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:39.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 114 -eq 0 ']' 2022-03-01T06:42:39.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:39.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:39.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:40.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 113 -eq 0 ']' 2022-03-01T06:42:40.540 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:40.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:40.866 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:41.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 112 -eq 0 ']' 2022-03-01T06:42:41.976 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:42.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:42.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:43.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 111 -eq 0 ']' 2022-03-01T06:42:43.202 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:43.533 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:43.533 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:44.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 110 -eq 0 ']' 2022-03-01T06:42:44.534 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:44.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:44.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:45.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 109 -eq 0 ']' 2022-03-01T06:42:45.882 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:46.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:46.213 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:47.215 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 108 -eq 0 ']' 2022-03-01T06:42:47.217 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:47.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:47.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:48.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 107 -eq 0 ']' 2022-03-01T06:42:48.538 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:48.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:48.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:49.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 106 -eq 0 ']' 2022-03-01T06:42:49.874 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:50.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:50.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:51.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 105 -eq 0 ']' 2022-03-01T06:42:51.207 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:51.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:51.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:52.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 104 -eq 0 ']' 2022-03-01T06:42:52.539 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:52.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:52.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:53.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 103 -eq 0 ']' 2022-03-01T06:42:53.871 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:54.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:54.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:55.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 102 -eq 0 ']' 2022-03-01T06:42:55.203 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:55.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:55.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:56.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 101 -eq 0 ']' 2022-03-01T06:42:56.538 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:56.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:56.874 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:57.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 100 -eq 0 ']' 2022-03-01T06:42:57.875 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:58.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:58.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:42:59.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 99 -eq 0 ']' 2022-03-01T06:42:59.208 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:42:59.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:42:59.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:00.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 98 -eq 0 ']' 2022-03-01T06:43:00.541 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:00.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:00.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:01.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 97 -eq 0 ']' 2022-03-01T06:43:01.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:02.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:02.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:03.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 96 -eq 0 ']' 2022-03-01T06:43:03.208 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:03.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:03.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:04.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 95 -eq 0 ']' 2022-03-01T06:43:04.542 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:04.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:04.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:05.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 94 -eq 0 ']' 2022-03-01T06:43:05.877 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:06.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:06.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:07.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 93 -eq 0 ']' 2022-03-01T06:43:07.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:07.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:07.541 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:08.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 92 -eq 0 ']' 2022-03-01T06:43:08.542 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:08.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:08.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:09.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 91 -eq 0 ']' 2022-03-01T06:43:09.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:10.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:10.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:11.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 90 -eq 0 ']' 2022-03-01T06:43:11.204 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:11.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:11.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:12.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 89 -eq 0 ']' 2022-03-01T06:43:12.539 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:12.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:12.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:13.869 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 88 -eq 0 ']' 2022-03-01T06:43:13.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:14.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:14.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:15.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 87 -eq 0 ']' 2022-03-01T06:43:15.202 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:15.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:15.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:16.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 86 -eq 0 ']' 2022-03-01T06:43:16.529 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:16.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:16.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:17.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 85 -eq 0 ']' 2022-03-01T06:43:17.864 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:18.195 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:18.195 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:19.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 84 -eq 0 ']' 2022-03-01T06:43:19.198 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:19.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:19.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:20.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 83 -eq 0 ']' 2022-03-01T06:43:20.530 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:20.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:20.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:21.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 82 -eq 0 ']' 2022-03-01T06:43:21.866 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:22.196 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:22.196 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:23.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 81 -eq 0 ']' 2022-03-01T06:43:23.197 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:23.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:23.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:24.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 80 -eq 0 ']' 2022-03-01T06:43:24.531 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:24.863 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:24.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:25.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 79 -eq 0 ']' 2022-03-01T06:43:25.865 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:26.196 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:26.196 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:27.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 78 -eq 0 ']' 2022-03-01T06:43:27.197 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:27.527 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:27.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:28.528 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 77 -eq 0 ']' 2022-03-01T06:43:28.529 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:28.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:28.859 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:29.860 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 76 -eq 0 ']' 2022-03-01T06:43:29.861 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:30.217 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:30.218 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:31.219 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 75 -eq 0 ']' 2022-03-01T06:43:31.219 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:31.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:31.550 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:32.551 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 74 -eq 0 ']' 2022-03-01T06:43:32.552 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:32.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:32.881 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:33.882 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 73 -eq 0 ']' 2022-03-01T06:43:33.882 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:34.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:34.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:35.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 72 -eq 0 ']' 2022-03-01T06:43:35.209 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:35.538 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:35.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:36.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 71 -eq 0 ']' 2022-03-01T06:43:36.540 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:36.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:36.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:37.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 70 -eq 0 ']' 2022-03-01T06:43:37.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:38.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:38.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:39.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 69 -eq 0 ']' 2022-03-01T06:43:39.202 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:39.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:39.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:40.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 68 -eq 0 ']' 2022-03-01T06:43:40.535 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:40.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:40.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:41.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 67 -eq 0 ']' 2022-03-01T06:43:41.868 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:42.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:42.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:43.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 66 -eq 0 ']' 2022-03-01T06:43:43.201 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:43.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:43.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:44.532 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 65 -eq 0 ']' 2022-03-01T06:43:44.533 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:44.866 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:44.866 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:45.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 64 -eq 0 ']' 2022-03-01T06:43:45.868 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:46.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:46.197 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:47.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 63 -eq 0 ']' 2022-03-01T06:43:47.199 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:47.529 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:47.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:48.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 62 -eq 0 ']' 2022-03-01T06:43:48.531 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:48.864 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:48.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:49.865 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 61 -eq 0 ']' 2022-03-01T06:43:49.866 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:50.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:50.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:51.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 60 -eq 0 ']' 2022-03-01T06:43:51.204 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:51.533 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:51.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:52.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 59 -eq 0 ']' 2022-03-01T06:43:52.535 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:52.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:52.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:53.869 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 58 -eq 0 ']' 2022-03-01T06:43:53.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:54.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:54.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:55.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 57 -eq 0 ']' 2022-03-01T06:43:55.202 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:55.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:55.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:56.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 56 -eq 0 ']' 2022-03-01T06:43:56.535 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:56.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:56.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:57.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 55 -eq 0 ']' 2022-03-01T06:43:57.877 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:58.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:58.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:43:59.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 54 -eq 0 ']' 2022-03-01T06:43:59.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:43:59.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:43:59.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:00.544 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 53 -eq 0 ']' 2022-03-01T06:44:00.544 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:00.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:00.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:01.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 52 -eq 0 ']' 2022-03-01T06:44:01.878 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:02.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:02.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:03.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 51 -eq 0 ']' 2022-03-01T06:44:03.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:03.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:03.544 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:04.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 50 -eq 0 ']' 2022-03-01T06:44:04.545 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:04.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:04.873 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:05.874 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 49 -eq 0 ']' 2022-03-01T06:44:05.875 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:06.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:06.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:07.207 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 48 -eq 0 ']' 2022-03-01T06:44:07.207 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:07.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:07.539 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:08.540 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 47 -eq 0 ']' 2022-03-01T06:44:08.540 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:08.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:08.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:09.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 46 -eq 0 ']' 2022-03-01T06:44:09.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:10.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:10.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:11.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 45 -eq 0 ']' 2022-03-01T06:44:11.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:11.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:11.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:12.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 44 -eq 0 ']' 2022-03-01T06:44:12.536 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:12.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:12.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:13.868 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 43 -eq 0 ']' 2022-03-01T06:44:13.869 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:14.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:14.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:15.199 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 42 -eq 0 ']' 2022-03-01T06:44:15.200 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:15.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:15.530 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:16.531 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 41 -eq 0 ']' 2022-03-01T06:44:16.532 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:16.866 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:16.866 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:17.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 40 -eq 0 ']' 2022-03-01T06:44:17.867 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:18.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:18.198 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:19.199 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 39 -eq 0 ']' 2022-03-01T06:44:19.200 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:19.533 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:19.533 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:20.534 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 38 -eq 0 ']' 2022-03-01T06:44:20.535 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:20.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:20.929 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:21.867 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 37 -eq 0 ']' 2022-03-01T06:44:21.868 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:22.200 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:22.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:23.201 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 36 -eq 0 ']' 2022-03-01T06:44:23.202 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:23.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:23.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:24.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 35 -eq 0 ']' 2022-03-01T06:44:24.537 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:24.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:24.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:25.872 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 34 -eq 0 ']' 2022-03-01T06:44:25.873 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:26.202 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:26.203 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:27.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 33 -eq 0 ']' 2022-03-01T06:44:27.205 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:27.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:27.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:28.537 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 32 -eq 0 ']' 2022-03-01T06:44:28.538 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:28.870 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:28.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:29.871 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 31 -eq 0 ']' 2022-03-01T06:44:29.871 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:30.204 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:30.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:31.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 30 -eq 0 ']' 2022-03-01T06:44:31.206 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:31.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:31.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:32.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 29 -eq 0 ']' 2022-03-01T06:44:32.544 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:32.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:32.875 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:33.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 28 -eq 0 ']' 2022-03-01T06:44:33.876 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:34.208 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:34.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:35.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 27 -eq 0 ']' 2022-03-01T06:44:35.210 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:35.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:35.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:36.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 26 -eq 0 ']' 2022-03-01T06:44:36.544 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:36.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:36.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:37.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 25 -eq 0 ']' 2022-03-01T06:44:37.879 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:38.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:38.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:39.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 24 -eq 0 ']' 2022-03-01T06:44:39.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:39.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:39.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:40.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 23 -eq 0 ']' 2022-03-01T06:44:40.546 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:40.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:40.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:41.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 22 -eq 0 ']' 2022-03-01T06:44:41.879 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:42.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:42.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:43.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 21 -eq 0 ']' 2022-03-01T06:44:43.211 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:43.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:43.542 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:44.543 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 20 -eq 0 ']' 2022-03-01T06:44:44.543 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:44.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:44.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:45.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 19 -eq 0 ']' 2022-03-01T06:44:45.879 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:46.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:46.209 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:47.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 18 -eq 0 ']' 2022-03-01T06:44:47.210 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:47.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:47.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:48.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 17 -eq 0 ']' 2022-03-01T06:44:48.546 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:48.877 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:48.878 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:49.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 16 -eq 0 ']' 2022-03-01T06:44:49.879 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:50.210 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:50.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:51.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 15 -eq 0 ']' 2022-03-01T06:44:51.212 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:51.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:51.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:52.545 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 14 -eq 0 ']' 2022-03-01T06:44:52.546 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:52.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:52.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:53.876 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 13 -eq 0 ']' 2022-03-01T06:44:53.877 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:54.211 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:54.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:55.212 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 12 -eq 0 ']' 2022-03-01T06:44:55.213 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:55.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:55.546 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:56.547 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 11 -eq 0 ']' 2022-03-01T06:44:56.548 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:56.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:56.879 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:57.880 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 10 -eq 0 ']' 2022-03-01T06:44:57.881 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:58.205 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:58.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:44:59.206 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 9 -eq 0 ']' 2022-03-01T06:44:59.207 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:44:59.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:44:59.535 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:00.536 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 8 -eq 0 ']' 2022-03-01T06:45:00.537 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:00.929 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:00.929 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:01.930 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 7 -eq 0 ']' 2022-03-01T06:45:01.930 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:02.281 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:02.281 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:03.282 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 6 -eq 0 ']' 2022-03-01T06:45:03.283 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:03.614 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:03.614 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:04.615 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 5 -eq 0 ']' 2022-03-01T06:45:04.615 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:04.945 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:04.945 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:05.946 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 4 -eq 0 ']' 2022-03-01T06:45:05.947 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:06.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:06.280 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:07.280 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 3 -eq 0 ']' 2022-03-01T06:45:07.281 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:07.612 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:07.612 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:08.613 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 2 -eq 0 ']' 2022-03-01T06:45:08.614 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:08.946 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:08.946 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:09.947 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 1 -eq 0 ']' 2022-03-01T06:45:09.948 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2022-03-01T06:45:10.275 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2022-03-01T06:45:10.275 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2022-03-01T06:45:11.276 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 0 -eq 0 ']' 2022-03-01T06:45:11.277 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: return 1 2022-03-01T06:45:11.277 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: return 1 2022-03-01T06:45:11.277 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2022-03-01T06:45:11.278 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2022-03-01T06:45:11.278 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2022-03-01T06:45:11.279 INFO:tasks.workunit.client.0.smithi169.stdout:writing non-divergent object existing_4 2022-03-01T06:45:11.279 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2022-03-01T09:32:06.527 INFO:tasks.workunit.client.0.smithi169.stderr:Terminated 2022-03-01T09:32:06.551 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:1: TEST_divergent_3: teardown td/divergent-priors 1 2022-03-01T09:32:06.551 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2022-03-01T09:32:06.552 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2022-03-01T09:32:06.552 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2022-03-01T09:32:06.553 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2022-03-01T09:32:06.553 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2022-03-01T09:32:06.553 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2022-03-01T09:32:06.553 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2022-03-01T09:32:06.554 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2022-03-01T09:32:06.634 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2022-03-01T09:32:06.635 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2022-03-01T09:32:06.635 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2022-03-01T09:32:06.636 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2022-03-01T09:32:06.637 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2022-03-01T09:32:06.638 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2022-03-01T09:32:06.638 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2022-03-01T09:32:06.641 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T09:32:06.641 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2022-03-01T09:32:06.642 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2022-03-01T09:32:06.642 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-01T09:32:06.643 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2022-03-01T09:32:06.644 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: cores=yes 2022-03-01T09:32:06.644 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:182: teardown: '[' -n '' ']' 2022-03-01T09:32:06.644 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' yes = yes -o 1 = 1 ']' 2022-03-01T09:32:06.645 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2022-03-01T09:32:06.645 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2022-03-01T09:32:06.645 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/divergent-priors/mgr.x.log td/divergent-priors/mon.a.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log /home/ubuntu/cephtest/archive/log 2022-03-01T09:32:06.646 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2022-03-01T09:32:06.651 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2022-03-01T09:32:06.652 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2022-03-01T09:32:06.652 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31520 2022-03-01T09:32:06.652 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31520 2022-03-01T09:32:06.653 INFO:tasks.workunit.client.0.smithi169.stdout:ERROR: Failure due to cores found 2022-03-01T09:32:06.654 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' yes = yes ']' 2022-03-01T09:32:06.654 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:201: teardown: echo 'ERROR: Failure due to cores found' 2022-03-01T09:32:06.654 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:202: teardown: '[' -n '' ']' 2022-03-01T09:32:06.654 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:205: teardown: return 1 2022-03-01T09:32:06.655 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2022-03-01T10:22:06.683 INFO:tasks.workunit.client.0.smithi169.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-03-01T10:22:06.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2022-03-01T10:22:06.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2022-03-01T10:22:06.720 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2022-03-01T10:22:06.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2022-03-01T10:22:06.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2022-03-01T10:22:06.721 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2022-03-01T10:22:06.721 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2022-03-01T10:22:06.722 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2022-03-01T10:22:06.722 INFO:tasks.workunit.client.0.smithi169.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2022-03-01T10:22:06.722 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2022-03-01T10:22:06.722 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2022-03-01T10:22:06.723 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2022-03-01T10:22:06.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2022-03-01T10:22:06.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2022-03-01T10:22:06.749 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2022-03-01T10:22:06.750 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2022-03-01T10:22:06.750 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2022-03-01T10:27:06.839 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:27:06.836+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T10:32:06.839 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:32:06.837+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T10:37:06.841 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:37:06.838+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T10:42:06.840 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:42:06.838+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T10:47:06.840 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:47:06.839+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T10:52:06.841 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:52:06.838+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T10:57:06.842 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T10:57:06.839+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T11:02:06.842 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T11:02:06.840+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T11:07:06.843 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T11:07:06.840+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T11:12:06.843 INFO:tasks.workunit.client.0.smithi169.stderr:2022-03-01T11:12:06.840+0000 7f502bb90700 0 monclient(hunting): authenticate timed out after 300 2022-03-01T11:12:06.844 INFO:tasks.workunit.client.0.smithi169.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-03-01T11:12:06.845 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids= 2022-03-01T11:12:06.846 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2022-03-01T11:12:06.846 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2022-03-01T11:12:06.847 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2022-03-01T11:12:06.847 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2022-03-01T12:02:06.948 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test == 0 2022-03-01T12:02:06.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1587: test: ==: unary operator expected 2022-03-01T12:02:06.977 INFO:tasks.workunit.client.0.smithi169.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2022-03-01T12:02:06.978 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2022-03-01T12:02:06.978 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2022-03-01T12:02:06.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2022-03-01T12:02:06.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2022-03-01T12:02:06.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2022-03-01T12:02:06.979 INFO:tasks.workunit.client.0.smithi169.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2022-03-01T12:51:10.171 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-01T12:51:10.198 DEBUG:teuthology.task.console_log:Killing console logger for smithi169 2022-03-01T12:51:10.199 DEBUG:teuthology.exit:Finished running handlers