2024-01-10T06:34:10.384 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-10T06:34:10.384 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi/7510458 --description rados/standalone/{supported-random-distro$/{centos_8} workloads/misc} --name yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi/7510458/orig.config.yaml 2024-01-10T06:34:10.406 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-10T06:34:10.483 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi/7510458 branch: quincy description: rados/standalone/{supported-random-distro$/{centos_8} workloads/misc} email: yweinste@redhat.com first_in_suite: false job_id: '7510458' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6441305d08929b8a72c0322a5c268c64b4a99c65 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6441305d08929b8a72c0322a5c268c64b4a99c65 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-63937-quincy sha1: affad57d5ae9c6b2f7579f7a454052bf9428828f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6252 sha1: 6441305d08929b8a72c0322a5c268c64b4a99c65 sleep_before_teardown: 0 subset: 111/12000 suite: rados suite_branch: wip-63937-quincy suite_path: /home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: affad57d5ae9c6b2f7579f7a454052bf9428828f targets: smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMaxpaq1R0hS1TYBzZx6wJyUceMDMiSVe4L408tP2oyOEUIeTZmBCcO+KxNQfFBjWiw7K/PM07hw/izD6tbKyUc= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-09_20:55:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-10T06:34:10.483 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa; will attempt to use it 2024-01-10T06:34:10.484 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks 2024-01-10T06:34:10.484 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-10T06:34:10.486 INFO:teuthology.task.internal:Checking packages... 2024-01-10T06:34:10.509 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6441305d08929b8a72c0322a5c268c64b4a99c65' 2024-01-10T06:34:10.510 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-10T06:34:10.510 INFO:teuthology.packaging:ref: None 2024-01-10T06:34:10.510 INFO:teuthology.packaging:tag: None 2024-01-10T06:34:10.510 INFO:teuthology.packaging:branch: quincy 2024-01-10T06:34:10.510 INFO:teuthology.packaging:sha1: 6441305d08929b8a72c0322a5c268c64b4a99c65 2024-01-10T06:34:10.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-01-10T06:34:10.737 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-268.g6441305d 2024-01-10T06:34:10.739 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-10T06:34:10.747 INFO:teuthology.task.internal:no buildpackages task found 2024-01-10T06:34:10.747 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-10T06:34:10.771 INFO:teuthology.task.internal:Saving configuration 2024-01-10T06:34:10.781 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-10T06:34:10.792 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-10T06:34:10.816 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi/7510458', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-10 06:28:57.938045', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMaxpaq1R0hS1TYBzZx6wJyUceMDMiSVe4L408tP2oyOEUIeTZmBCcO+KxNQfFBjWiw7K/PM07hw/izD6tbKyUc='} 2024-01-10T06:34:10.816 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-10T06:34:10.827 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-10T06:34:10.827 INFO:teuthology.run_tasks:Running task console_log... 2024-01-10T06:34:10.873 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcade2e3ee0>, signals=[15]) 2024-01-10T06:34:10.874 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-10T06:34:10.881 INFO:teuthology.task.internal:Opening connections... 2024-01-10T06:34:10.882 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2024-01-10T06:34:10.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:34:10.960 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-10T06:34:10.968 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2024-01-10T06:34:10.990 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2024-01-10T06:34:10.991 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-10T06:34:11.052 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-10T06:34:11.053 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-10T06:34:11.053 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2024-01-10T06:34:11.074 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-10T06:34:11.083 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-10T06:34:11.090 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-10T06:34:11.091 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2024-01-10T06:34:11.112 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-10T06:34:11.119 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-10T06:34:11.120 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2024-01-10T06:34:11.187 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-10T06:34:11.221 INFO:teuthology.run_tasks:Running task kernel... 2024-01-10T06:34:11.235 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-10T06:34:11.235 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-10T06:34:11.236 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-10T06:34:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T06:34:11.256 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-01-10T06:34:11.316 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-489.el8.x86_64 2024-01-10T06:34:11.316 INFO:teuthology.task.kernel:Running kernel on smithi057: 4.18.0-489.el8.x86_64 2024-01-10T06:34:11.316 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2024-01-10T06:34:14.926 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - AppStream 33 MB/s | 35 MB 00:01 2024-01-10T06:34:23.668 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 58 MB 00:01 2024-01-10T06:34:32.843 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-01-10T06:34:49.550 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras common packages 430 B/s | 6.9 kB 00:16 2024-01-10T06:34:49.893 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-01-10T06:34:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-01-10T06:34:55.660 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 415 kB/s | 24 kB 00:00 2024-01-10T06:34:59.655 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-10T06:34:59.655 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-10T06:34:59.970 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:34:59.970 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:Install 3 Packages 2024-01-10T06:34:59.971 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:34:59.972 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 90 M 2024-01-10T06:34:59.972 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 96 M 2024-01-10T06:34:59.972 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:35:00.836 INFO:teuthology.orchestra.run.smithi057.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2024-01-10T06:35:01.895 INFO:teuthology.orchestra.run.smithi057.stdout:(2/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-01-10T06:35:02.678 INFO:teuthology.orchestra.run.smithi057.stdout:(3/3): kernel-core-4.18.0-532.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2024-01-10T06:35:02.679 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:35:02.679 INFO:teuthology.orchestra.run.smithi057.stdout:Total 33 MB/s | 90 MB 00:02 2024-01-10T06:35:03.394 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:35:03.860 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:35:03.860 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:35:05.171 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:35:05.204 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:35:08.130 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:35:08.649 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-10T06:35:12.402 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-10T06:35:12.799 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-10T06:35:16.476 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-10T06:35:16.777 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-10T06:35:40.144 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-10T06:35:40.169 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-10T06:35:40.523 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-10T06:35:40.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-10T06:35:40.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-10T06:35:41.343 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-10T06:35:41.343 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:35:41.343 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:35:41.343 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-10T06:35:41.343 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-10T06:35:41.344 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:35:41.344 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:35:41.830 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-01-10T06:35:42.059 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-10T06:35:42.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-10T06:35:42.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-10T06:35:42.059 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-10T06:35:42.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-10T06:35:42.060 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-10T06:35:42.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2024-01-10T06:35:42.060 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-01-10T06:35:42.279 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-10T06:35:42.280 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-10T06:35:42.280 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.02 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 129.el8 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Group : System Environment/Base 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 9265586 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-10T06:35:42.315 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:Relocations : (not relocatable) 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : CentOS BuildSystem 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:35:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2024-01-10T06:35:42.317 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-10T06:35:42.317 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 4.18.0-532.el8.x86_64 2024-01-10T06:35:42.318 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-10T06:35:43.581 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2024-01-10T06:35:45.776 INFO:teuthology.orchestra.run.smithi057.stderr:done 2024-01-10T06:35:45.777 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-01-10T06:35:45.808 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.6W9KKviYOF 2024-01-10T06:35:45.808 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6W9KKviYOF 2024-01-10T06:35:45.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.6W9KKviYOF 2024-01-10T06:35:46.021 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.6W9KKviYOF is 6KB 2024-01-10T06:35:46.069 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.6W9KKviYOF 2024-01-10T06:35:46.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2024-01-10T06:35:46.161 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-10T06:35:46.161 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-10T06:35:46.161 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-10T06:35:46.162 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-10T06:35:46.163 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-10T06:35:46.336 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2024-01-10T06:36:16.368 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-10T06:36:16.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:36:16.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:36:34.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-01-10T06:36:43.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:36:43.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:36:46.816 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-01-10T06:36:58.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:36:58.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:37:01.888 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-01-10T06:37:16.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:37:16.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:37:35.420 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-01-10T06:37:53.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:37:53.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:37:53.768 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-01-10T06:37:54.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:37:54.093 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-532.el8.x86_64"... 2024-01-10T06:37:54.094 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-01-10T06:37:54.111 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-532.el8.x86_64 2024-01-10T06:37:54.111 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-10T06:37:54.111 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-10T06:37:54.111 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-10T06:37:55.112 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-10T06:37:55.112 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-10T06:37:55.839 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2024-01-10T06:37:55.877 DEBUG:teuthology.parallel:result is None 2024-01-10T06:37:55.878 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-10T06:37:55.887 INFO:teuthology.task.internal:Creating test directory... 2024-01-10T06:37:55.887 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-10T06:37:56.356 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-10T06:37:56.362 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-10T06:37:56.373 INFO:teuthology.task.internal:Creating archive directory... 2024-01-10T06:37:56.373 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-10T06:37:56.443 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-10T06:37:56.452 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-10T06:37:56.452 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-10T06:37:56.497 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:37:56.508 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:37:56.510 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-10T06:37:56.518 INFO:teuthology.task.internal:Configuring sudo... 2024-01-10T06:37:56.518 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-10T06:37:56.547 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-10T06:37:56.557 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-10T06:37:56.557 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-10T06:37:56.605 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-10T06:37:56.692 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-10T06:37:56.788 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-10T06:37:56.829 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-10T06:37:56.876 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:37:56.876 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-10T06:37:56.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2024-01-10T06:37:57.167 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-10T06:37:57.386 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-10T06:37:57.394 INFO:teuthology.task.internal:Starting timer... 2024-01-10T06:37:57.394 INFO:teuthology.run_tasks:Running task pcp... 2024-01-10T06:37:57.406 INFO:teuthology.run_tasks:Running task selinux... 2024-01-10T06:37:57.417 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-10T06:37:57.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2024-01-10T06:37:57.513 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2024-01-10T06:37:57.515 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-10T06:37:57.515 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2024-01-10T06:37:57.554 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2024-01-10T06:37:57.555 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive'} 2024-01-10T06:37:57.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-10T06:37:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T06:37:57.623 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-10T06:37:57.623 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2024-01-10T06:37:57.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-10T06:37:57.705 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-10T06:37:57.710 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-10T06:37:57.821 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-10T06:37:57.837 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-10T06:37:57.838 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-10T06:43:55.313 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com')] 2024-01-10T06:43:55.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:43:55.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-10T06:43:55.393 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-01-10T06:43:55.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-01-10T06:43:55.468 INFO:teuthology.run_tasks:Running task clock... 2024-01-10T06:43:55.480 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-10T06:43:55.480 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-10T06:43:55.480 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-10T06:43:55.542 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-10T06:43:55.557 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-10T06:43:55.623 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2024-01-10T06:43:55.635 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2024-01-10T06:43:55.648 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-10T06:43:55.662 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-10T06:43:55.731 INFO:teuthology.orchestra.run.smithi057.stderr:bash: ntpq: command not found 2024-01-10T06:43:55.734 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-10T06:43:55.734 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-01-10T06:43:55.734 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-10T06:43:55.734 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-10T06:43:55.734 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-10T06:43:55.734 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-10T06:43:55.735 INFO:teuthology.run_tasks:Running task install... 2024-01-10T06:43:55.744 DEBUG:teuthology.task.install:project ceph 2024-01-10T06:43:55.745 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6441305d08929b8a72c0322a5c268c64b4a99c65'}} 2024-01-10T06:43:55.745 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '6441305d08929b8a72c0322a5c268c64b4a99c65'} 2024-01-10T06:43:55.745 INFO:teuthology.task.install:Using flavor: default 2024-01-10T06:43:55.750 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-10T06:43:55.750 INFO:teuthology.task.install:extra packages: [] 2024-01-10T06:43:55.751 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6441305d08929b8a72c0322a5c268c64b4a99c65', 'tag': None, 'wait_for_package': False} 2024-01-10T06:43:55.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=6441305d08929b8a72c0322a5c268c64b4a99c65 2024-01-10T06:43:55.894 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/8/flavors/default/ 2024-01-10T06:43:55.894 INFO:teuthology.task.install.rpm:Package version is 17.2.7-268.g6441305d 2024-01-10T06:43:56.215 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-10T06:43:56.215 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:43:56.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-10T06:43:56.250 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-10T06:43:56.252 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6441305d08929b8a72c0322a5c268c64b4a99c65/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-10T06:43:56.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-10T06:43:56.408 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-10T06:43:56.482 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2024-01-10T06:43:56.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2024-01-10T06:43:56.789 INFO:teuthology.orchestra.run.smithi057.stdout:76 files removed 2024-01-10T06:43:56.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw 2024-01-10T06:43:58.048 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 98 kB/s | 78 kB 00:00 2024-01-10T06:43:58.757 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 14 kB/s | 9.1 kB 00:00 2024-01-10T06:43:59.299 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 2.8 kB/s | 1.4 kB 00:00 2024-01-10T06:44:01.066 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - AppStream 20 MB/s | 35 MB 00:01 2024-01-10T06:44:09.657 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - BaseOS 32 MB/s | 58 MB 00:01 2024-01-10T06:44:19.250 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-01-10T06:44:19.776 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2024-01-10T06:44:20.744 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - PowerTools 8.3 MB/s | 6.7 MB 00:00 2024-01-10T06:44:22.279 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-01-10T06:44:23.282 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-01-10T06:44:27.572 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 515 kB/s | 24 kB 00:00 2024-01-10T06:44:32.528 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:44:32.531 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:44:32.531 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:44:32.531 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:17.2.7-268.g6441305d.el8 ceph 17 M 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:17.2.7-268.g6441305d.el8 ceph 3.5 M 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:17.2.7-268.g6441305d.el8 ceph 3.6 M 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:17.2.7-268.g6441305d.el8 ceph 5.2 M 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:17.2.7-268.g6441305d.el8 ceph 23 M 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:17.2.7-268.g6441305d.el8 ceph 24 k 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:17.2.7-268.g6441305d.el8 ceph 796 k 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-10T06:44:32.532 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:17.2.7-268.g6441305d.el8 ceph 506 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:17.2.7-268.g6441305d.el8 ceph 5.6 M 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse x86_64 2:17.2.7-268.g6441305d.el8 ceph 45 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:17.2.7-268.g6441305d.el8 ceph 115 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:17.2.7-268.g6441305d.el8 ceph 190 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:17.2.7-268.g6441305d.el8 ceph 371 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:17.2.7-268.g6441305d.el8 ceph 358 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:17.2.7-268.g6441305d.el8 ceph 107 k 2024-01-10T06:44:32.533 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout:Install 21 Packages 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 2 Packages 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 64 M 2024-01-10T06:44:32.534 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:44:34.140 INFO:teuthology.orchestra.run.smithi057.stdout:(1/23): ceph-base-17.2.7-268.g6441305d.el8.x86_ 3.5 MB/s | 5.2 MB 00:01 2024-01-10T06:44:34.248 INFO:teuthology.orchestra.run.smithi057.stdout:(2/23): ceph-selinux-17.2.7-268.g6441305d.el8.x 225 kB/s | 24 kB 00:00 2024-01-10T06:44:34.724 INFO:teuthology.orchestra.run.smithi057.stdout:(3/23): libcephfs2-17.2.7-268.g6441305d.el8.x86 1.6 MB/s | 796 kB 00:00 2024-01-10T06:44:34.924 INFO:teuthology.orchestra.run.smithi057.stdout:(4/23): libradosstriper1-17.2.7-268.g6441305d.e 2.5 MB/s | 506 kB 00:00 2024-01-10T06:44:36.424 INFO:teuthology.orchestra.run.smithi057.stdout:(5/23): librgw2-17.2.7-268.g6441305d.el8.x86_64 3.7 MB/s | 5.6 MB 00:01 2024-01-10T06:44:36.533 INFO:teuthology.orchestra.run.smithi057.stdout:(6/23): python3-ceph-argparse-17.2.7-268.g64413 413 kB/s | 45 kB 00:00 2024-01-10T06:44:36.642 INFO:teuthology.orchestra.run.smithi057.stdout:(7/23): python3-ceph-common-17.2.7-268.g6441305 1.0 MB/s | 115 kB 00:00 2024-01-10T06:44:37.150 INFO:teuthology.orchestra.run.smithi057.stdout:(8/23): python3-cephfs-17.2.7-268.g6441305d.el8 373 kB/s | 190 kB 00:00 2024-01-10T06:44:37.526 INFO:teuthology.orchestra.run.smithi057.stdout:(9/23): ceph-common-17.2.7-268.g6441305d.el8.x8 4.8 MB/s | 23 MB 00:04 2024-01-10T06:44:37.551 INFO:teuthology.orchestra.run.smithi057.stdout:(10/23): python3-rados-17.2.7-268.g6441305d.el8 928 kB/s | 371 kB 00:00 2024-01-10T06:44:37.643 INFO:teuthology.orchestra.run.smithi057.stdout:(11/23): python3-rgw-17.2.7-268.g6441305d.el8.x 1.1 MB/s | 107 kB 00:00 2024-01-10T06:44:37.710 INFO:teuthology.orchestra.run.smithi057.stdout:(12/23): python3-rbd-17.2.7-268.g6441305d.el8.x 1.9 MB/s | 358 kB 00:00 2024-01-10T06:44:37.931 INFO:teuthology.orchestra.run.smithi057.stdout:(13/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.3 MB/s | 677 kB 00:00 2024-01-10T06:44:37.961 INFO:teuthology.orchestra.run.smithi057.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-01-10T06:44:37.986 INFO:teuthology.orchestra.run.smithi057.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 808 kB/s | 44 kB 00:00 2024-01-10T06:44:38.111 INFO:teuthology.orchestra.run.smithi057.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2024-01-10T06:44:38.195 INFO:teuthology.orchestra.run.smithi057.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-01-10T06:44:38.237 INFO:teuthology.orchestra.run.smithi057.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2024-01-10T06:44:38.262 INFO:teuthology.orchestra.run.smithi057.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 390 kB/s | 59 kB 00:00 2024-01-10T06:44:38.738 INFO:teuthology.orchestra.run.smithi057.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 3.5 MB/s | 1.7 MB 00:00 2024-01-10T06:44:39.638 INFO:teuthology.orchestra.run.smithi057.stdout:(21/23): librbd1-17.2.7-268.g6441305d.el8.x86_6 4.0 MB/s | 3.6 MB 00:00 2024-01-10T06:44:40.080 INFO:teuthology.orchestra.run.smithi057.stdout:(22/23): librados2-17.2.7-268.g6441305d.el8.x86 1.9 MB/s | 3.5 MB 00:01 2024-01-10T06:44:43.947 INFO:teuthology.orchestra.run.smithi057.stdout:(23/23): ceph-radosgw-17.2.7-268.g6441305d.el8. 1.5 MB/s | 17 MB 00:11 2024-01-10T06:44:43.950 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:44:43.951 INFO:teuthology.orchestra.run.smithi057.stdout:Total 5.6 MB/s | 64 MB 00:11 2024-01-10T06:44:43.994 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:44:44.044 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:44:44.044 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:44:44.482 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:44:44.485 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:44:45.669 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:44:45.860 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-01-10T06:44:46.039 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-10T06:44:46.474 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-10T06:44:46.686 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:17.2.7-268.g6441305d.el8.x86_64 3/25 2024-01-10T06:44:46.781 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:17.2.7-268.g6441305d.el8.x86_64 3/25 2024-01-10T06:44:47.073 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:17.2.7-268.g6441305d.el8.x86_64 4/25 2024-01-10T06:44:47.236 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:17.2.7-268.g6441305d.el8.x86_64 5/25 2024-01-10T06:44:47.688 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:17.2.7-268.g6441305d.el8.x86_64 5/25 2024-01-10T06:44:47.894 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:17.2.7-268.g6441305d.el8.x86_64 6/25 2024-01-10T06:44:48.133 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:17.2.7-268.g6441305d.el8.x86_64 6/25 2024-01-10T06:44:48.307 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-01-10T06:44:48.541 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2024-01-10T06:44:49.274 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-01-10T06:44:49.461 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:17.2.7-268.g6441305d.el8.x86_64 10/25 2024-01-10T06:44:49.525 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:17.2.7-268.g6441305d.el8.x86_64 10/25 2024-01-10T06:44:50.129 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:17.2.7-268.g6441305d.el8.x 11/25 2024-01-10T06:44:50.323 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:17.2.7-268.g6441305d.el8.x86_64 12/25 2024-01-10T06:44:50.557 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:17.2.7-268.g6441305d.el8.x86_64 13/25 2024-01-10T06:44:50.785 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:17.2.7-268.g6441305d.el8.x86_64 14/25 2024-01-10T06:44:50.910 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:17.2.7-268.g6441305d.el8.x86_64 15/25 2024-01-10T06:44:51.008 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:17.2.7-268.g6441305d.el8.x86_64 15/25 2024-01-10T06:44:51.231 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-01-10T06:44:51.428 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-01-10T06:44:51.625 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-01-10T06:44:51.827 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:17.2.7-268.g6441305d.el8.x86 19/25 2024-01-10T06:44:54.720 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el8.x86_64 20/25 2024-01-10T06:44:55.069 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:17.2.7-268.g6441305d.el8.x86_64 20/25 2024-01-10T06:44:55.684 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el8.x86_64 20/25 2024-01-10T06:44:55.949 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:17.2.7-268.g6441305d.el8.x86_64 21/25 2024-01-10T06:44:56.056 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:17.2.7-268.g6441305d.el8.x86_64 21/25 2024-01-10T06:44:56.243 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:17.2.7-268.g6441305d.el8.x86_64 22/25 2024-01-10T06:45:16.478 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:17.2.7-268.g6441305d.el8.x86_64 22/25 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /dev 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /run 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /dev 2024-01-10T06:45:16.479 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:18.271 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:17.2.7-268.g6441305d.el8.x86_64 23/25 2024-01-10T06:45:18.332 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-268.g6441305d.el8.x86_64 23/25 2024-01-10T06:45:18.332 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-10T06:45:18.742 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-10T06:45:18.742 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-10T06:45:19.446 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:17.2.7-268.g6441305d.el8.x86_64 1/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:17.2.7-268.g6441305d.el8.x86_64 2/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:17.2.7-268.g6441305d.el8.x86_64 3/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:17.2.7-268.g6441305d.el8.x86_64 4/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:17.2.7-268.g6441305d.el8.x86_64 5/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:17.2.7-268.g6441305d.el8.x86_64 6/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:17.2.7-268.g6441305d.el8.x86_64 7/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:17.2.7-268.g6441305d.el8.x 8/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:17.2.7-268.g6441305d.el8.x86 9/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:17.2.7-268.g6441305d.el8.x86_64 10/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:17.2.7-268.g6441305d.el8.x86_64 11/25 2024-01-10T06:45:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:17.2.7-268.g6441305d.el8.x86_64 12/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:17.2.7-268.g6441305d.el8.x86_64 13/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:17.2.7-268.g6441305d.el8.x86_64 22/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-01-10T06:45:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:17.2.7-268.g6441305d.el8.x86_64 24/25 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.749 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.750 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-10T06:45:20.751 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.751 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.751 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:20.751 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-10T06:45:20.751 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:20.751 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:45:21.102 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-test 2024-01-10T06:45:21.992 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:54 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:45:23.814 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:45:23.815 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:45:23.815 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:17.2.7-268.g6441305d.el8 ceph 52 M 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:17.2.7-268.g6441305d.el8 ceph 178 k 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:45:23.816 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:45:23.817 INFO:teuthology.orchestra.run.smithi057.stdout:Install 6 Packages 2024-01-10T06:45:23.817 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:23.817 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 53 M 2024-01-10T06:45:23.817 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 234 M 2024-01-10T06:45:23.817 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:45:24.089 INFO:teuthology.orchestra.run.smithi057.stdout:(1/6): libcephsqlite-17.2.7-268.g6441305d.el8.x 952 kB/s | 178 kB 00:00 2024-01-10T06:45:24.190 INFO:teuthology.orchestra.run.smithi057.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 707 kB/s | 203 kB 00:00 2024-01-10T06:45:24.332 INFO:teuthology.orchestra.run.smithi057.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2024-01-10T06:45:24.415 INFO:teuthology.orchestra.run.smithi057.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 847 kB/s | 71 kB 00:00 2024-01-10T06:45:24.441 INFO:teuthology.orchestra.run.smithi057.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 534 kB/s | 187 kB 00:00 2024-01-10T06:45:26.491 INFO:teuthology.orchestra.run.smithi057.stdout:(6/6): ceph-test-17.2.7-268.g6441305d.el8.x86_6 20 MB/s | 52 MB 00:02 2024-01-10T06:45:26.493 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:45:26.493 INFO:teuthology.orchestra.run.smithi057.stdout:Total 20 MB/s | 53 MB 00:02 2024-01-10T06:45:26.537 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:45:26.559 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:45:26.559 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:45:26.896 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:45:26.899 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:45:27.400 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:45:27.627 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-10T06:45:27.851 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-10T06:45:28.038 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-10T06:45:28.147 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-10T06:45:28.330 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-10T06:45:28.471 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:17.2.7-268.g6441305d.el8.x86_64 5/6 2024-01-10T06:45:33.754 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:17.2.7-268.g6441305d.el8.x86_64 5/6 2024-01-10T06:45:34.095 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:17.2.7-268.g6441305d.el8.x86_64 6/6 2024-01-10T06:45:34.430 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-test-2:17.2.7-268.g6441305d.el8.x86_64 6/6 2024-01-10T06:45:34.430 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:17.2.7-268.g6441305d.el8.x86_64 1/6 2024-01-10T06:45:34.430 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:17.2.7-268.g6441305d.el8.x86_64 2/6 2024-01-10T06:45:34.430 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-10T06:45:34.430 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-10T06:45:34.430 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-10T06:45:35.223 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-10T06:45:35.223 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-8.el8.x86_64 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:35.224 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:45:35.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph 2024-01-10T06:45:36.370 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:09 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:45:38.207 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:17.2.7-268.g6441305d.el8 ceph 6.4 k 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:17.2.7-268.g6441305d.el8 ceph 2.3 M 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:17.2.7-268.g6441305d.el8 ceph 1.6 M 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 238 k 2024-01-10T06:45:38.211 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:17.2.7-268.g6441305d.el8 ceph 3.9 M 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:17.2.7-268.g6441305d.el8 ceph 17 M 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-10T06:45:38.212 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-10T06:45:38.213 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout:Install 32 Packages 2024-01-10T06:45:38.214 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:38.215 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 28 M 2024-01-10T06:45:38.215 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 110 M 2024-01-10T06:45:38.215 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:45:38.819 INFO:teuthology.orchestra.run.smithi057.stdout:(1/32): ceph-17.2.7-268.g6441305d.el8.x86_64.rp 21 kB/s | 6.4 kB 00:00 2024-01-10T06:45:39.427 INFO:teuthology.orchestra.run.smithi057.stdout:(2/32): ceph-mgr-17.2.7-268.g6441305d.el8.x86_6 1.7 MB/s | 1.6 MB 00:00 2024-01-10T06:45:39.503 INFO:teuthology.orchestra.run.smithi057.stdout:(3/32): ceph-mds-17.2.7-268.g6441305d.el8.x86_6 2.3 MB/s | 2.3 MB 00:00 2024-01-10T06:45:39.601 INFO:teuthology.orchestra.run.smithi057.stdout:(4/32): ceph-mon-17.2.7-268.g6441305d.el8.x86_6 5.0 MB/s | 3.9 MB 00:00 2024-01-10T06:45:39.628 INFO:teuthology.orchestra.run.smithi057.stdout:(5/32): ceph-mgr-modules-core-17.2.7-268.g64413 1.9 MB/s | 238 kB 00:00 2024-01-10T06:45:39.745 INFO:teuthology.orchestra.run.smithi057.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 333 kB/s | 39 kB 00:00 2024-01-10T06:45:39.770 INFO:teuthology.orchestra.run.smithi057.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 926 kB/s | 157 kB 00:00 2024-01-10T06:45:39.854 INFO:teuthology.orchestra.run.smithi057.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 683 kB/s | 57 kB 00:00 2024-01-10T06:45:39.880 INFO:teuthology.orchestra.run.smithi057.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 773 kB/s | 103 kB 00:00 2024-01-10T06:45:40.030 INFO:teuthology.orchestra.run.smithi057.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.5 MB/s | 457 kB 00:00 2024-01-10T06:45:40.239 INFO:teuthology.orchestra.run.smithi057.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 694 kB/s | 249 kB 00:00 2024-01-10T06:45:40.455 INFO:teuthology.orchestra.run.smithi057.stdout:(12/32): ceph-osd-17.2.7-268.g6441305d.el8.x86_ 16 MB/s | 17 MB 00:01 2024-01-10T06:45:40.481 INFO:teuthology.orchestra.run.smithi057.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 451 kB/s | 109 kB 00:00 2024-01-10T06:45:40.582 INFO:teuthology.orchestra.run.smithi057.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-01-10T06:45:41.031 INFO:teuthology.orchestra.run.smithi057.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 76 kB/s | 44 kB 00:00 2024-01-10T06:45:41.140 INFO:teuthology.orchestra.run.smithi057.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 281 kB/s | 185 kB 00:00 2024-01-10T06:45:41.190 INFO:teuthology.orchestra.run.smithi057.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 289 kB/s | 173 kB 00:00 2024-01-10T06:45:41.249 INFO:teuthology.orchestra.run.smithi057.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-01-10T06:45:41.291 INFO:teuthology.orchestra.run.smithi057.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-01-10T06:45:41.358 INFO:teuthology.orchestra.run.smithi057.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2024-01-10T06:45:41.400 INFO:teuthology.orchestra.run.smithi057.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 542 kB/s | 59 kB 00:00 2024-01-10T06:45:41.508 INFO:teuthology.orchestra.run.smithi057.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2024-01-10T06:45:41.616 INFO:teuthology.orchestra.run.smithi057.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 173 kB/s | 19 kB 00:00 2024-01-10T06:45:41.642 INFO:teuthology.orchestra.run.smithi057.stdout:(24/32): python3-cherrypy-18.4.0-1.el8.noarch.r 630 kB/s | 384 kB 00:00 2024-01-10T06:45:41.725 INFO:teuthology.orchestra.run.smithi057.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-01-10T06:45:41.751 INFO:teuthology.orchestra.run.smithi057.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2024-01-10T06:45:41.784 INFO:teuthology.orchestra.run.smithi057.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 663 kB/s | 283 kB 00:00 2024-01-10T06:45:41.818 INFO:teuthology.orchestra.run.smithi057.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2024-01-10T06:45:42.010 INFO:teuthology.orchestra.run.smithi057.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 446 kB/s | 85 kB 00:00 2024-01-10T06:45:42.102 INFO:teuthology.orchestra.run.smithi057.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 728 kB/s | 255 kB 00:00 2024-01-10T06:45:42.118 INFO:teuthology.orchestra.run.smithi057.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-01-10T06:45:42.173 INFO:teuthology.orchestra.run.smithi057.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 647 kB/s | 251 kB 00:00 2024-01-10T06:45:42.178 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:45:42.178 INFO:teuthology.orchestra.run.smithi057.stdout:Total 7.2 MB/s | 28 MB 00:03 2024-01-10T06:45:42.265 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:45:42.305 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:45:42.305 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:45:42.612 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:45:42.615 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:45:43.966 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:45:44.204 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-10T06:45:44.570 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-10T06:45:44.811 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-10T06:45:45.017 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-10T06:45:45.232 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-10T06:45:45.457 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-10T06:45:45.681 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-10T06:45:45.900 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-10T06:45:46.138 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-10T06:45:46.382 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-10T06:45:46.629 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-10T06:45:46.895 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-10T06:45:47.324 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-10T06:45:47.723 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-10T06:45:47.927 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-10T06:45:48.194 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-10T06:45:48.438 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-10T06:45:48.832 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-10T06:45:49.152 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-10T06:45:49.381 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-10T06:45:49.600 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-10T06:45:49.912 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-10T06:45:50.235 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-10T06:45:50.543 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-10T06:45:50.907 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-10T06:45:51.274 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-10T06:45:52.065 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el8.n 27/32 2024-01-10T06:45:52.766 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:17.2.7-268.g6441305d.el8.x86_64 28/32 2024-01-10T06:45:54.364 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:17.2.7-268.g6441305d.el8.x86_64 28/32 2024-01-10T06:45:54.532 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:17.2.7-268.g6441305d.el8.x86_64 29/32 2024-01-10T06:45:54.961 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:17.2.7-268.g6441305d.el8.x86_64 29/32 2024-01-10T06:45:55.159 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:17.2.7-268.g6441305d.el8.x86_64 30/32 2024-01-10T06:45:55.388 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:17.2.7-268.g6441305d.el8.x86_64 30/32 2024-01-10T06:45:55.568 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:17.2.7-268.g6441305d.el8.x86_64 31/32 2024-01-10T06:45:55.586 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:17.2.7-268.g6441305d.el8.x86_64 31/32 2024-01-10T06:45:55.709 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:17.2.7-268.g6441305d.el8.x86_64 32/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-2:17.2.7-268.g6441305d.el8.x86_64 32/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:17.2.7-268.g6441305d.el8.x86_64 1/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:17.2.7-268.g6441305d.el8.x86_64 2/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:17.2.7-268.g6441305d.el8.x86_64 3/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:17.2.7-268.g6441305d.el8.x86_64 4/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:17.2.7-268.g6441305d.el8.x86_64 5/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el8.n 6/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-10T06:45:56.165 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-10T06:45:56.166 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-10T06:45:56.167 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-2.6-1.el8.noarch 2024-01-10T06:45:57.515 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:45:57.516 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:45:57.747 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-base 2024-01-10T06:45:58.636 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:31 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:00.222 INFO:teuthology.orchestra.run.smithi057.stdout:Package ceph-base-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:46:00.472 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:00.474 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:46:00.474 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:00.589 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install cephadm 2024-01-10T06:46:01.480 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:34 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:03.315 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:03.316 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:03.316 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:46:03.316 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:03.316 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:03.316 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 80 k 2024-01-10T06:46:03.316 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 80 k 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 364 k 2024-01-10T06:46:03.317 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:04.124 INFO:teuthology.orchestra.run.smithi057.stdout:cephadm-17.2.7-268.g6441305d.el8.noarch.rpm 100 kB/s | 80 kB 00:00 2024-01-10T06:46:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:Total 99 kB/s | 80 kB 00:00 2024-01-10T06:46:04.125 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:04.135 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:04.135 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:04.138 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:04.141 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:04.660 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:05.477 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:46:05.644 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:46:05.887 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:46:06.644 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:46:06.644 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:06.644 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:46:06.644 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:06.645 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:06.645 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:06.866 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-immutable-object-cache 2024-01-10T06:46:07.762 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:40 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:09.600 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-268.g6441305d.el8 ceph 161 k 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:46:09.601 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:09.602 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 161 k 2024-01-10T06:46:09.602 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 490 k 2024-01-10T06:46:09.602 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:10.087 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-immutable-object-cache-17.2.7-268.g6441305 333 kB/s | 161 kB 00:00 2024-01-10T06:46:10.087 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:10.087 INFO:teuthology.orchestra.run.smithi057.stdout:Total 332 kB/s | 161 kB 00:00 2024-01-10T06:46:10.088 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:10.100 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:10.100 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:10.154 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:10.157 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:10.285 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:10.474 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-10T06:46:10.905 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-10T06:46:11.642 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-10T06:46:11.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:11.643 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:46:11.643 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:46:11.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:11.643 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:11.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr 2024-01-10T06:46:12.742 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:45 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:14.321 INFO:teuthology.orchestra.run.smithi057.stdout:Package ceph-mgr-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:46:14.576 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:14.578 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:46:14.578 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:14.687 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-dashboard 2024-01-10T06:46:15.577 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:48 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:17.395 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:17.396 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:17.396 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:46:17.396 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:17.397 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:17.397 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 1.7 M 2024-01-10T06:46:17.397 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:46:17.397 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 23 k 2024-01-10T06:46:17.398 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 14 k 2024-01-10T06:46:17.398 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-10T06:46:17.398 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-10T06:46:17.398 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-10T06:46:17.398 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:17.398 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:17.399 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:17.399 INFO:teuthology.orchestra.run.smithi057.stdout:Install 6 Packages 2024-01-10T06:46:17.399 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:17.400 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 2.0 M 2024-01-10T06:46:17.400 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 8.0 M 2024-01-10T06:46:17.400 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:18.256 INFO:teuthology.orchestra.run.smithi057.stdout:(1/6): ceph-grafana-dashboards-17.2.7-268.g6441 79 kB/s | 23 kB 00:00 2024-01-10T06:46:18.340 INFO:teuthology.orchestra.run.smithi057.stdout:(2/6): ceph-prometheus-alerts-17.2.7-268.g64413 38 kB/s | 14 kB 00:00 2024-01-10T06:46:18.432 INFO:teuthology.orchestra.run.smithi057.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 366 kB/s | 34 kB 00:00 2024-01-10T06:46:18.457 INFO:teuthology.orchestra.run.smithi057.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 213 kB/s | 43 kB 00:00 2024-01-10T06:46:18.482 INFO:teuthology.orchestra.run.smithi057.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 3.8 MB/s | 196 kB 00:00 2024-01-10T06:46:19.699 INFO:teuthology.orchestra.run.smithi057.stdout:(6/6): ceph-mgr-dashboard-17.2.7-268.g6441305d. 998 kB/s | 1.7 MB 00:01 2024-01-10T06:46:19.702 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:19.702 INFO:teuthology.orchestra.run.smithi057.stdout:Total 888 kB/s | 2.0 MB 00:02 2024-01-10T06:46:19.714 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:19.724 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:19.724 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:19.817 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:19.820 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:20.118 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:20.567 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-10T06:46:20.810 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-10T06:46:21.025 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-10T06:46:21.671 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el8.no 4/6 2024-01-10T06:46:21.671 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el8.n 5/6 2024-01-10T06:46:21.752 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el8.noarch 6/6 2024-01-10T06:46:22.054 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el8.noarch 6/6 2024-01-10T06:46:22.054 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el8.n 1/6 2024-01-10T06:46:22.054 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el8.noarch 2/6 2024-01-10T06:46:22.054 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el8.no 3/6 2024-01-10T06:46:22.054 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-10T06:46:22.054 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-10T06:46:22.820 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:22.821 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:22.989 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-10T06:46:23.890 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:56 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:25.724 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:25.726 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2024-01-10T06:46:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:46:25.726 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2024-01-10T06:46:25.726 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 7.4 M 2024-01-10T06:46:25.727 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:46:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-10T06:46:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-10T06:46:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-10T06:46:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-10T06:46:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-10T06:46:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-10T06:46:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-10T06:46:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:25.728 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:25.728 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2024-01-10T06:46:25.729 INFO:teuthology.orchestra.run.smithi057.stdout:Install 8 Packages 2024-01-10T06:46:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:25.730 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 36 M 2024-01-10T06:46:25.730 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 210 M 2024-01-10T06:46:25.730 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:26.790 INFO:teuthology.orchestra.run.smithi057.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 5.6 MB/s | 4.7 MB 00:00 2024-01-10T06:46:26.940 INFO:teuthology.orchestra.run.smithi057.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.9 MB 00:00 2024-01-10T06:46:26.999 INFO:teuthology.orchestra.run.smithi057.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2024-01-10T06:46:27.116 INFO:teuthology.orchestra.run.smithi057.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-01-10T06:46:27.441 INFO:teuthology.orchestra.run.smithi057.stdout:(5/8): ceph-mgr-diskprediction-local-17.2.7-268 5.0 MB/s | 7.4 MB 00:01 2024-01-10T06:46:27.766 INFO:teuthology.orchestra.run.smithi057.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 18 MB/s | 14 MB 00:00 2024-01-10T06:46:27.791 INFO:teuthology.orchestra.run.smithi057.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 493 kB/s | 172 kB 00:00 2024-01-10T06:46:27.825 INFO:teuthology.orchestra.run.smithi057.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 910 kB/s | 645 kB 00:00 2024-01-10T06:46:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:Total 17 MB/s | 36 MB 00:02 2024-01-10T06:46:28.069 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:28.088 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:28.088 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:28.488 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:28.491 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:29.106 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:29.262 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-10T06:46:29.381 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-10T06:46:29.554 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-10T06:46:30.174 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-10T06:46:30.412 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-10T06:46:31.063 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-10T06:46:31.237 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-10T06:46:31.993 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-10T06:46:32.346 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-10T06:46:32.611 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-10T06:46:34.525 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-10T06:46:35.863 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-10T06:46:36.086 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d 8/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d 8/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d 1/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-10T06:46:36.421 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-10T06:46:36.422 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-10T06:46:37.213 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-10T06:46:37.213 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:37.214 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:37.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-rook 2024-01-10T06:46:38.239 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:11 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:40.073 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:40.074 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 47 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-10T06:46:40.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout:Install 13 Packages 2024-01-10T06:46:40.076 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:40.077 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 1.9 M 2024-01-10T06:46:40.077 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 25 M 2024-01-10T06:46:40.077 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:40.471 INFO:teuthology.orchestra.run.smithi057.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 157 kB/s | 20 kB 00:00 2024-01-10T06:46:40.488 INFO:teuthology.orchestra.run.smithi057.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 184 kB/s | 27 kB 00:00 2024-01-10T06:46:40.605 INFO:teuthology.orchestra.run.smithi057.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 949 kB/s | 126 kB 00:00 2024-01-10T06:46:40.622 INFO:teuthology.orchestra.run.smithi057.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 823 kB/s | 110 kB 00:00 2024-01-10T06:46:40.698 INFO:teuthology.orchestra.run.smithi057.stdout:(5/13): ceph-mgr-rook-17.2.7-268.g6441305d.el8. 134 kB/s | 47 kB 00:00 2024-01-10T06:46:40.739 INFO:teuthology.orchestra.run.smithi057.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 364 kB/s | 43 kB 00:00 2024-01-10T06:46:40.840 INFO:teuthology.orchestra.run.smithi057.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 662 kB/s | 155 kB 00:00 2024-01-10T06:46:40.998 INFO:teuthology.orchestra.run.smithi057.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 110 kB/s | 33 kB 00:00 2024-01-10T06:46:41.024 INFO:teuthology.orchestra.run.smithi057.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 57 kB/s | 16 kB 00:00 2024-01-10T06:46:41.257 INFO:teuthology.orchestra.run.smithi057.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 195 kB/s | 81 kB 00:00 2024-01-10T06:46:41.283 INFO:teuthology.orchestra.run.smithi057.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 248 kB/s | 64 kB 00:00 2024-01-10T06:46:41.416 INFO:teuthology.orchestra.run.smithi057.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 387 kB/s | 61 kB 00:00 2024-01-10T06:46:41.566 INFO:teuthology.orchestra.run.smithi057.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2024-01-10T06:46:41.566 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:41.567 INFO:teuthology.orchestra.run.smithi057.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-01-10T06:46:41.629 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:41.645 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:41.645 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:41.738 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:41.741 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:42.298 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:42.615 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-10T06:46:42.862 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-10T06:46:43.114 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-10T06:46:43.360 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-10T06:46:43.577 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-10T06:46:43.853 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-10T06:46:44.156 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-10T06:46:44.418 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-10T06:46:45.574 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-10T06:46:45.925 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-10T06:46:46.142 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-10T06:46:46.329 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-10T06:46:46.534 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:17.2.7-268.g6441305d.el8.noarch 13/13 2024-01-10T06:46:46.858 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-268.g6441305d.el8.noarch 13/13 2024-01-10T06:46:46.858 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:17.2.7-268.g6441305d.el8.noarch 1/13 2024-01-10T06:46:46.858 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-10T06:46:46.858 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-10T06:46:46.858 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-10T06:46:46.859 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-10T06:46:47.700 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:47.701 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:47.885 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-cephadm 2024-01-10T06:46:48.770 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:21 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:50.627 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 121 k 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-10T06:46:50.628 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout:Install 5 Packages 2024-01-10T06:46:50.629 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:50.630 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 6.0 M 2024-01-10T06:46:50.630 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 26 M 2024-01-10T06:46:50.630 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:51.098 INFO:teuthology.orchestra.run.smithi057.stdout:(1/5): ceph-mgr-cephadm-17.2.7-268.g6441305d.el 573 kB/s | 121 kB 00:00 2024-01-10T06:46:51.184 INFO:teuthology.orchestra.run.smithi057.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2024-01-10T06:46:51.349 INFO:teuthology.orchestra.run.smithi057.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2024-01-10T06:46:51.541 INFO:teuthology.orchestra.run.smithi057.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 168 kB/s | 60 kB 00:00 2024-01-10T06:46:51.566 INFO:teuthology.orchestra.run.smithi057.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2024-01-10T06:46:51.568 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:51.569 INFO:teuthology.orchestra.run.smithi057.stdout:Total 6.3 MB/s | 6.0 MB 00:00 2024-01-10T06:46:51.625 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:51.638 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:51.638 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:51.749 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:51.753 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:52.078 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:52.451 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-01-10T06:46:53.558 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-01-10T06:46:53.954 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-01-10T06:46:54.238 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2024-01-10T06:46:54.458 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el8.noarch 5/5 2024-01-10T06:46:54.760 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el8.noarch 5/5 2024-01-10T06:46:54.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el8.noarch 1/5 2024-01-10T06:46:54.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-01-10T06:46:54.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2024-01-10T06:46:54.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-01-10T06:46:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-01-10T06:46:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:55.543 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:55.544 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:46:55.655 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-fuse 2024-01-10T06:46:56.549 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:29 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:46:58.383 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:17.2.7-268.g6441305d.el8 ceph 875 k 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout:Install 2 Packages 2024-01-10T06:46:58.384 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:46:58.385 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 958 k 2024-01-10T06:46:58.385 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 3.0 M 2024-01-10T06:46:58.385 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:46:58.740 INFO:teuthology.orchestra.run.smithi057.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 508 kB/s | 83 kB 00:00 2024-01-10T06:46:59.272 INFO:teuthology.orchestra.run.smithi057.stdout:(2/2): ceph-fuse-17.2.7-268.g6441305d.el8.x86_6 1.2 MB/s | 875 kB 00:00 2024-01-10T06:46:59.273 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:46:59.273 INFO:teuthology.orchestra.run.smithi057.stdout:Total 1.1 MB/s | 958 kB 00:00 2024-01-10T06:46:59.285 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:46:59.309 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:46:59.309 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:46:59.407 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:46:59.410 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:46:59.596 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:46:59.880 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-10T06:47:00.091 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:17.2.7-268.g6441305d.el8.x86_64 2/2 2024-01-10T06:47:00.487 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-fuse-2:17.2.7-268.g6441305d.el8.x86_64 2/2 2024-01-10T06:47:00.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:17.2.7-268.g6441305d.el8.x86_64 1/2 2024-01-10T06:47:01.217 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-10T06:47:01.217 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:01.218 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:01.218 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:17.2.7-268.g6441305d.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-10T06:47:01.218 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:01.218 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:01.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-volume 2024-01-10T06:47:02.411 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:35 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:04.242 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:04.243 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:04.243 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-01-10T06:47:04.243 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:04.243 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:17.2.7-268.g6441305d.el8 ceph-noarch 267 k 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 267 k 2024-01-10T06:47:04.244 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 1.3 M 2024-01-10T06:47:04.245 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:47:04.612 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-volume-17.2.7-268.g6441305d.el8.noarch.rpm 725 kB/s | 267 kB 00:00 2024-01-10T06:47:04.612 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:47:04.613 INFO:teuthology.orchestra.run.smithi057.stdout:Total 723 kB/s | 267 kB 00:00 2024-01-10T06:47:04.613 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:47:04.624 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:47:04.625 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:47:04.665 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:47:04.668 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:47:04.885 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:47:05.182 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:47:05.597 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:47:06.362 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:17.2.7-268.g6441305d.el8.noarch 1/1 2024-01-10T06:47:06.362 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:06.362 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:06.362 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:17.2.7-268.g6441305d.el8.noarch 2024-01-10T06:47:06.362 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:06.362 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:06.565 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librados-devel 2024-01-10T06:47:07.457 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:40 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:17.2.7-268.g6441305d.el8 ceph 130 k 2024-01-10T06:47:09.284 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:09.285 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:47:09.285 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:09.285 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:47:09.285 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:09.285 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 130 k 2024-01-10T06:47:09.285 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 464 k 2024-01-10T06:47:09.286 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:47:09.750 INFO:teuthology.orchestra.run.smithi057.stdout:librados-devel-17.2.7-268.g6441305d.el8.x86_64. 280 kB/s | 130 kB 00:00 2024-01-10T06:47:09.750 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:47:09.750 INFO:teuthology.orchestra.run.smithi057.stdout:Total 279 kB/s | 130 kB 00:00 2024-01-10T06:47:09.751 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:47:09.757 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:47:09.757 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:47:09.815 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:47:09.818 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:47:09.931 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:47:10.197 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:10.489 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados-devel-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:11.204 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:11.205 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:11.205 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:11.205 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:47:11.205 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:11.205 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:11.449 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install libcephfs2 2024-01-10T06:47:12.345 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:45 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:13.934 INFO:teuthology.orchestra.run.smithi057.stdout:Package libcephfs2-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:14.184 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:14.185 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:14.186 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:14.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install libcephfs-devel 2024-01-10T06:47:15.183 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:48 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:17.010 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:17.2.7-268.g6441305d.el8 ceph 25 k 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:47:17.011 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:17.012 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 25 k 2024-01-10T06:47:17.012 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 109 k 2024-01-10T06:47:17.012 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:47:17.153 INFO:teuthology.orchestra.run.smithi057.stdout:libcephfs-devel-17.2.7-268.g6441305d.el8.x86_64 177 kB/s | 25 kB 00:00 2024-01-10T06:47:17.153 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:47:17.153 INFO:teuthology.orchestra.run.smithi057.stdout:Total 175 kB/s | 25 kB 00:00 2024-01-10T06:47:17.154 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:47:17.157 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:47:17.157 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:47:17.159 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:47:17.162 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:47:17.233 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:47:17.454 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:17.720 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:18.432 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:18.432 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:18.432 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:18.433 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:47:18.433 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:18.433 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:18.557 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librados2 2024-01-10T06:47:19.463 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:52 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:21.059 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:21.310 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:21.311 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:21.424 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librbd1 2024-01-10T06:47:22.309 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:55 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:23.909 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:24.163 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:24.164 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:24.164 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:24.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rados 2024-01-10T06:47:25.179 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:02:58 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:26.747 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rados-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:26.999 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:27.000 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:27.000 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:27.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rgw 2024-01-10T06:47:28.002 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:03:01 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:29.576 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rgw-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:29.830 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:29.831 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:29.831 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:29.935 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-cephfs 2024-01-10T06:47:30.832 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:03:03 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:32.408 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-cephfs-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:32.664 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:32.665 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:32.666 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:32.774 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rbd 2024-01-10T06:47:33.682 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:03:06 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:35.254 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rbd-2:17.2.7-268.g6441305d.el8.x86_64 is already installed. 2024-01-10T06:47:35.511 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:35.512 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-01-10T06:47:35.512 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:35.622 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-fuse 2024-01-10T06:47:36.513 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:03:09 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:38.349 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:17.2.7-268.g6441305d.el8 ceph 91 k 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:47:38.350 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:38.351 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:47:38.351 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:38.351 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 91 k 2024-01-10T06:47:38.351 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 264 k 2024-01-10T06:47:38.351 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:47:38.816 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-fuse-17.2.7-268.g6441305d.el8.x86_64.rpm 195 kB/s | 91 kB 00:00 2024-01-10T06:47:38.817 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:47:38.818 INFO:teuthology.orchestra.run.smithi057.stdout:Total 194 kB/s | 91 kB 00:00 2024-01-10T06:47:38.818 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:47:38.828 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:47:38.828 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:47:38.882 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:47:38.886 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:47:39.005 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:47:39.234 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:39.534 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-fuse-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:40.299 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:40.299 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:40.300 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:40.300 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:47:40.300 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:40.300 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:40.531 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-mirror 2024-01-10T06:47:41.423 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:03:14 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:43.255 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:43.256 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:43.256 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-01-10T06:47:43.256 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:17.2.7-268.g6441305d.el8 ceph 3.5 M 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 3.5 M 2024-01-10T06:47:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 16 M 2024-01-10T06:47:43.258 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:47:44.610 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-mirror-17.2.7-268.g6441305d.el8.x86_64.rpm 2.6 MB/s | 3.5 MB 00:01 2024-01-10T06:47:44.610 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:47:44.611 INFO:teuthology.orchestra.run.smithi057.stdout:Total 2.5 MB/s | 3.5 MB 00:01 2024-01-10T06:47:44.611 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:47:44.625 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:47:44.625 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:47:44.697 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:47:44.700 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:47:45.170 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:47:45.401 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:45.872 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:46.661 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:46.661 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:46.661 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:46.661 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:47:46.661 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:46.661 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:46.876 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-nbd 2024-01-10T06:47:47.771 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:03:20 ago on Wed 10 Jan 2024 06:44:27 AM UTC. 2024-01-10T06:47:49.606 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-01-10T06:47:49.607 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:49.607 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-01-10T06:47:49.607 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:49.607 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-01-10T06:47:49.607 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:17.2.7-268.g6441305d.el8 ceph 174 k 2024-01-10T06:47:49.607 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:49.608 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-01-10T06:47:49.608 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-01-10T06:47:49.608 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-01-10T06:47:49.608 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:49.608 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 174 k 2024-01-10T06:47:49.608 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 556 k 2024-01-10T06:47:49.609 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-01-10T06:47:50.120 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-nbd-17.2.7-268.g6441305d.el8.x86_64.rpm 340 kB/s | 174 kB 00:00 2024-01-10T06:47:50.120 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-01-10T06:47:50.120 INFO:teuthology.orchestra.run.smithi057.stdout:Total 339 kB/s | 174 kB 00:00 2024-01-10T06:47:50.121 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-01-10T06:47:50.133 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-01-10T06:47:50.133 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-01-10T06:47:50.188 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-01-10T06:47:50.191 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-01-10T06:47:50.311 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-01-10T06:47:50.515 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:50.815 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-nbd-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:51.596 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:17.2.7-268.g6441305d.el8.x86_64 1/1 2024-01-10T06:47:51.596 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:51.596 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-01-10T06:47:51.596 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:17.2.7-268.g6441305d.el8.x86_64 2024-01-10T06:47:51.596 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-01-10T06:47:51.597 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-01-10T06:47:51.834 DEBUG:teuthology.parallel:result is None 2024-01-10T06:47:51.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=6441305d08929b8a72c0322a5c268c64b4a99c65 2024-01-10T06:47:52.081 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-10T06:47:52.105 INFO:teuthology.orchestra.run.smithi057.stdout:17.2.7-268.g6441305d.el8 2024-01-10T06:47:52.105 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-268.g6441305d.el8 2024-01-10T06:47:52.106 INFO:teuthology.task.install:The correct ceph version 17.2.7-268.g6441305d is installed. 2024-01-10T06:47:52.107 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-10T06:47:52.107 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:47:52.108 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-10T06:47:52.175 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-10T06:47:52.175 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:47:52.176 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2024-01-10T06:47:52.243 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-10T06:47:52.311 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-10T06:47:52.311 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:47:52.311 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-10T06:47:52.378 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-10T06:47:52.445 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-10T06:47:52.446 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:47:52.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2024-01-10T06:47:52.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-10T06:47:52.579 INFO:teuthology.run_tasks:Running task workunit... 2024-01-10T06:47:52.593 INFO:tasks.workunit:Pulling workunits from ref affad57d5ae9c6b2f7579f7a454052bf9428828f 2024-01-10T06:47:52.594 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-10T06:47:52.594 INFO:tasks.workunit:timeout=3h 2024-01-10T06:47:52.594 INFO:tasks.workunit:cleanup=True 2024-01-10T06:47:52.594 DEBUG:teuthology.orchestra.run.smithi057:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-10T06:47:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T06:47:52.637 INFO:teuthology.orchestra.run.smithi057.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-10T06:47:52.638 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-01-10T06:47:52.695 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-01-10T06:47:52.695 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-01-10T06:47:52.753 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout affad57d5ae9c6b2f7579f7a454052bf9428828f 2024-01-10T06:47:52.811 INFO:tasks.workunit.client.0.smithi057.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr:Note: switching to 'affad57d5ae9c6b2f7579f7a454052bf9428828f'. 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr:state without impacting any branches by switching back to a branch. 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-10T06:48:38.012 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr: git switch -c 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr:Or undo this operation with: 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr: git switch - 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-01-10T06:48:38.013 INFO:tasks.workunit.client.0.smithi057.stderr:HEAD is now at affad57d5ae qa/suites: install pytest for pybind tasks 2024-01-10T06:48:38.016 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-10T06:48:38.075 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-01-10T06:48:38.076 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-10T06:48:38.133 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-01-10T06:48:38.134 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-01-10T06:48:38.134 DEBUG:teuthology.orchestra.run.smithi057:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-01-10T06:48:38.209 INFO:tasks.workunit.client.0.smithi057.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-01-10T06:48:38.217 INFO:tasks.workunit.client.0.smithi057.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-01-10T06:48:38.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T06:48:38.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/mclock-config 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-01-10T06:48:38.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-01-10T06:48:38.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none ' 2024-01-10T06:48:38.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-01-10T06:48:38.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-01-10T06:48:38.223 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-01-10T06:48:38.223 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_profile_builtin_to_custom 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_profile_custom_to_builtin 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_profile_disallow_builtin_params_modify 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_profile_disallow_builtin_params_override 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_recovery_limit_adjustment_mclock' 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:48:38.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:48:38.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:48:38.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:48:38.228 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:48:38.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:48:38.228 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:48:38.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:48:38.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:48:38.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:48:38.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:48:38.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:48:38.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:48:38.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:48:38.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:48:38.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:48:38.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:48:38.235 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:48:38.235 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:38.235 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:38.235 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:48:38.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:48:38.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:48:38.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:48:38.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:48:38.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:38.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:38.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:48:38.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:48:38.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-01-10T06:48:38.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-01-10T06:48:38.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:48:38.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:48:38.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-01-10T06:48:38.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-01-10T06:48:38.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-01-10T06:48:38.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:48:38.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:48:38.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:48:38.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:48:38.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:48:38.245 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:48:38.245 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:48:38.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:48:38.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:48:38.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:48:38.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:48:38.246 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:48:38.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:48:38.247 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:48:38.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:48:38.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:48:38.247 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:48:38.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:38.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:48:38.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:48:38.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:48:38.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-01-10T06:48:38.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-01-10T06:48:38.706 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:48:38.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:48:38.950 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:48:38.951 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:48:38.951 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:48:38.951 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:48:38.952 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:48:38.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get fsid 2024-01-10T06:48:39.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:48:39.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:48:39.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:48:39.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:48:39.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:48:39.065 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:48:39.066 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get mon_host 2024-01-10T06:48:39.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-01-10T06:48:39.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-01-10T06:48:39.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:48:39.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:48:39.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:48:39.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-01-10T06:48:39.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:40.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:48:40.053 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:48:40.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:48:40.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-01-10T06:48:40.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-01-10T06:48:40.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:48:40.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:48:40.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:48:40.080 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T06:48:40.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-01-10T06:48:40.082 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:48:40.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a67c66e-56b9-4b37-a995-8de0e6862dd0 2024-01-10T06:48:40.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6a67c66e-56b9-4b37-a995-8de0e6862dd0' 2024-01-10T06:48:40.082 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 6a67c66e-56b9-4b37-a995-8de0e6862dd0 2024-01-10T06:48:40.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:48:40.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIPZ5leSSaBRAAztKm5YIiI4zI0Z9JkDbkFQ== 2024-01-10T06:48:40.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIPZ5leSSaBRAAztKm5YIiI4zI0Z9JkDbkFQ=="}' 2024-01-10T06:48:40.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a67c66e-56b9-4b37-a995-8de0e6862dd0 -i td/mclock-config/0/new.json 2024-01-10T06:48:40.578 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:48:40.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-01-10T06:48:40.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDIPZ5leSSaBRAAztKm5YIiI4zI0Z9JkDbkFQ== --osd-uuid 6a67c66e-56b9-4b37-a995-8de0e6862dd0 2024-01-10T06:48:40.613 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.613+0000 7f894788d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:48:40.615 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.615+0000 7f894788d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:48:40.616 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.616+0000 7f894788d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:48:40.616 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.616+0000 7f894788d540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:48:40.616 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.616+0000 7f894788d540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:48:40.617 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.616+0000 7f894788d540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:48:40.777 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:40.777+0000 7f894788d540 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-01-10T06:48:46.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-01-10T06:48:46.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:48:46.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:48:46.864 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:48:46.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:48:47.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:48:47.129 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:48:47.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T06:48:47.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:48:47.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:48:47.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:48:47.154 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:47.154+0000 7fb47f4d0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:48:47.186 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:47.186+0000 7fb47f4d0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:48:47.187 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:47.187+0000 7fb47f4d0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:48:47.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:48:47.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:48:47.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:48:47.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:48:47.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:48:47.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:47.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:48:47.439 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:48:47.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:47.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:47.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:47.702+0000 7fb47f4d0540 -1 Falling back to public interface 2024-01-10T06:48:47.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:48.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:48.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:48.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:48:48.974 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:48:48.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:48.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:49.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:50.106 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:48:50.106+0000 7fb47f4d0540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:48:50.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:50.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:50.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:48:50.471 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:48:50.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:50.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:50.950 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:51.951 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:48:51.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:51.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:51.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:48:51.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:51.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:52.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:53.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:53.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:53.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:48:53.414 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:48:53.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:53.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:54.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:55.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:55.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:55.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:48:55.038 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:48:55.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:55.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:55.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:56.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:56.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:56.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:48:56.543 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:48:56.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:56.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:57.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:58.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:58.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:58.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:48:58.019 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:48:58.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:58.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:48:58.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:48:59.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:48:59.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:48:59.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:48:59.474 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:48:59.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:48:59.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:00.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:01.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:01.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:01.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:49:01.052 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:49:01.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:01.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:01.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:02.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:02.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:02.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:49:02.524 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:49:02.525 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:02.525 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:03.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:04.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:04.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:04.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:49:04.612 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:49:04.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:04.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:05.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:06.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:06.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:06.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:49:06.192 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:49:06.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:06.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:06.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:07.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:07.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:07.900 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:49:07.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:49:07.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:07.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:09.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:10.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:10.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:10.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:49:10.006 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:49:10.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:10.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:10.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:11.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:11.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:11.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:49:11.617 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:49:11.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:11.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:12.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:13.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:13.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:13.788 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:49:13.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:49:13.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:13.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:14.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:15.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:15.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:15.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:49:15.480 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:49:15.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:15.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:16.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:17.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:17.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:17.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T06:49:17.083 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T06:49:17.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:17.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:17.547 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37398,v1:127.0.0.1:6803/37398] [v2:127.0.0.1:6804/37398,v1:127.0.0.1:6805/37398] exists,up 6a67c66e-56b9-4b37-a995-8de0e6862dd0 2024-01-10T06:49:17.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:17.549 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:17.550 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:17.550 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T06:49:17.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-01-10T06:49:17.657 INFO:tasks.workunit.client.0.smithi057.stdout:osd_max_backfills: 1 2024-01-10T06:49:17.657 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-01-10T06:49:17.658 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-01-10T06:49:18.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-01-10T06:49:20.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-01-10T06:49:20.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:20.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:20.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:20.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:20.377 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:20.377 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:20.377 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:20.377 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:20.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:20.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T06:49:20.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-01-10T06:49:20.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-01-10T06:49:20.496 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-01-10T06:49:20.496 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:20.497 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok dump_recovery_reservations 2024-01-10T06:49:20.607 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-01-10T06:49:20.607 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:20.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:20.610 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok dump_recovery_reservations 2024-01-10T06:49:20.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-01-10T06:49:20.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-01-10T06:49:20.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-01-10T06:49:21.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-01-10T06:49:21.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-01-10T06:49:23.786 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-01-10T06:49:23.786 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:23.786 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:23.786 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:23.786 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:23.787 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:23.787 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:23.787 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:23.787 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:23.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:23.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T06:49:23.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-01-10T06:49:23.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-01-10T06:49:23.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-01-10T06:49:23.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:23.908 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:23.909 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok dump_recovery_reservations 2024-01-10T06:49:24.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-01-10T06:49:24.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-01-10T06:49:24.018 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-01-10T06:49:24.018 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:24.018 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:24.018 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:24.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok dump_recovery_reservations 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:49:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:49:24.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:49:24.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:49:24.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-01-10T06:49:24.929 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already down. 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:49:24.940 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:49:24.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:24.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-01-10T06:49:25.465 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/37398,v1:127.0.0.1:6803/37398] [v2:127.0.0.1:6804/37398,v1:127.0.0.1:6805/37398] exists 6a67c66e-56b9-4b37-a995-8de0e6862dd0 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:49:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:49:25.475 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-01-10T06:49:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-01-10T06:49:25.477 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T06:49:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T06:49:25.490 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:25.488+0000 7fc89bcdc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:25.492 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:25.490+0000 7fc89bcdc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:25.493 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:25.491+0000 7fc89bcdc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:25.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:49:25.957 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:49:25.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:25.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:26.008 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:26.007+0000 7fc89bcdc540 -1 Falling back to public interface 2024-01-10T06:49:26.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:27.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:27.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:27.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:49:27.453 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:49:27.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:27.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:27.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:28.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:28.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:28.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:49:28.913 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:49:28.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:28.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:29.135 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:29.134+0000 7fc89bcdc540 -1 osd.0 6 log_to_monitors true 2024-01-10T06:49:29.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:30.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:30.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:30.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:49:30.417 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:49:30.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:30.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:30.902 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/38642,v1:127.0.0.1:6803/38642] [v2:127.0.0.1:6804/38642,v1:127.0.0.1:6805/38642] exists,up 6a67c66e-56b9-4b37-a995-8de0e6862dd0 2024-01-10T06:49:30.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:49:30.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:49:30.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:49:30.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:30.906 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:30.907 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:30.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:30.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T06:49:31.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-01-10T06:49:31.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:31.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok dump_recovery_reservations 2024-01-10T06:49:31.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-01-10T06:49:31.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:49:31.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok dump_recovery_reservations 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:49:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:49:31.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:49:31.338 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:49:31.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:49:31.339 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:49:31.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:49:31.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:49:31.340 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:49:31.342 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:49:31.342 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:49:31.342 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:49:31.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:49:31.343 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:49:31.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:49:31.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:49:31.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:49:31.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:49:31.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:49:31.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:49:31.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:49:31.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:49:31.352 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:49:31.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:49:31.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:49:31.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:49:31.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:49:31.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:49:31.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:49:31.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:49:31.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:49:31.357 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:49:31.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:49:31.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:49:31.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:49:31.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:49:31.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.360 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:49:31.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:49:31.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:49:31.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-01-10T06:49:31.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:49:31.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:49:31.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:49:31.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:49:31.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:49:31.364 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:49:31.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:49:31.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:49:31.365 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:49:31.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:49:31.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:49:31.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:49:31.367 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:49:31.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:49:31.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:49:31.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:49:31.370 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:49:31.370 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.370 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:49:31.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:49:31.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:49:31.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:49:31.372 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:49:31.372 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.372 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:49:31.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:49:31.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:49:31.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:49:31.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:49:31.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-01-10T06:49:31.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-01-10T06:49:31.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:49:31.747 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:49:32.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:49:32.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:49:32.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:49:32.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:49:32.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:49:32.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:49:32.009 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:49:32.010 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get fsid 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:49:32.113 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:32.114 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:32.114 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:49:32.114 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:49:32.114 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get mon_host 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-01-10T06:49:32.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:49:33.100 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:49:33.100 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:49:33.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:49:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:49:33.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:49:33.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcb70976-ec02-48bd-a8a5-3baf0675d440 2024-01-10T06:49:33.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fcb70976-ec02-48bd-a8a5-3baf0675d440' 2024-01-10T06:49:33.131 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 fcb70976-ec02-48bd-a8a5-3baf0675d440 2024-01-10T06:49:33.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:49:33.143 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9PZ5lQDmBCBAAEZ1ru+eow+4FU894eka3xw== 2024-01-10T06:49:33.143 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9PZ5lQDmBCBAAEZ1ru+eow+4FU894eka3xw=="}' 2024-01-10T06:49:33.143 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcb70976-ec02-48bd-a8a5-3baf0675d440 -i td/mclock-config/0/new.json 2024-01-10T06:49:33.503 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:49:33.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-01-10T06:49:33.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQD9PZ5lQDmBCBAAEZ1ru+eow+4FU894eka3xw== --osd-uuid fcb70976-ec02-48bd-a8a5-3baf0675d440 2024-01-10T06:49:33.539 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.538+0000 7f3f35ced540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:33.541 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.540+0000 7f3f35ced540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:33.543 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.541+0000 7f3f35ced540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:33.543 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.541+0000 7f3f35ced540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:49:33.543 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.541+0000 7f3f35ced540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:49:33.543 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.541+0000 7f3f35ced540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:49:33.634 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:33.633+0000 7f3f35ced540 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-01-10T06:49:39.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-01-10T06:49:39.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:49:39.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:49:39.348 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:49:39.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:49:39.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:49:39.991 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:49:39.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T06:49:39.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:49:39.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:49:39.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:49:40.014 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:40.013+0000 7f052f4a8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:40.028 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:40.027+0000 7f052f4a8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:40.029 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:40.028+0000 7f052f4a8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:49:40.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:49:40.457 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:49:40.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:40.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:40.548 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:40.548+0000 7f052f4a8540 -1 Falling back to public interface 2024-01-10T06:49:40.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:41.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:41.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:41.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:49:41.976 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:49:41.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:41.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:42.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:43.099 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:49:43.098+0000 7f052f4a8540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:49:43.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:43.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:43.462 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:49:43.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:49:43.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:43.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:44.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:45.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:45.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:45.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:49:45.022 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:49:45.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:45.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:45.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:46.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:46.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:46.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:49:46.477 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:49:46.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:46.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:47.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:48.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:48.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:48.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:49:48.020 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:49:48.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:48.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:48.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:49.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:49.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:49.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:49:49.521 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:49:49.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:49.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:50.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:51.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:51.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:51.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:49:51.044 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:49:51.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:51.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:51.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:52.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:52.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:52.496 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:49:52.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:49:52.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:52.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:53.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:54.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:54.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:54.050 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:49:54.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:49:54.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:54.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:54.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:55.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:55.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:55.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:49:55.501 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:49:55.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:55.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:56.409 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:57.410 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:57.410 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:57.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:49:57.411 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:49:57.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:57.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:49:58.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:49:59.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:49:59.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:49:59.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:49:59.064 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:49:59.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:49:59.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:00.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:01.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:01.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:01.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:50:01.085 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:50:01.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:01.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:02.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:03.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:03.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:03.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:50:03.125 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:50:03.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:03.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:04.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:05.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:05.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:05.008 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:50:05.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:50:05.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:05.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:06.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:07.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:07.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:07.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:50:07.043 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:50:07.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:07.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:07.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:08.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:08.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:08.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:50:08.501 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:50:08.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:08.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:08.957 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/40392,v1:127.0.0.1:6803/40392] [v2:127.0.0.1:6804/40392,v1:127.0.0.1:6805/40392] exists,up fcb70976-ec02-48bd-a8a5-3baf0675d440 2024-01-10T06:50:08.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:50:08.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:50:08.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:50:08.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-01-10T06:50:09.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-01-10T06:50:09.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-01-10T06:50:09.435 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-01-10T06:50:09.435 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-01-10T06:50:09.435 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-01-10T06:50:09.436 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:50:09.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-01-10T06:50:09.543 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-01-10T06:50:09.543 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-01-10T06:50:09.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-01-10T06:50:09.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-01-10T06:50:09.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-01-10T06:50:09.668 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:50:09.668 INFO:tasks.workunit.client.0.smithi057.stdout: "success": "" 2024-01-10T06:50:09.668 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-01-10T06:50:09.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:50:09.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-01-10T06:50:09.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-01-10T06:50:09.787 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-01-10T06:50:09.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-01-10T06:50:09.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-01-10T06:50:09.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-01-10T06:50:09.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-01-10T06:50:09.788 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-01-10T06:50:09.789 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:50:09.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-01-10T06:50:09.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-01-10T06:50:09.895 INFO:tasks.workunit.client.0.smithi057.stdout:client_res = 0.300000 2024-01-10T06:50:09.896 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-01-10T06:50:09.896 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-01-10T06:50:09.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-01-10T06:50:09.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-01-10T06:50:09.897 INFO:tasks.workunit.client.0.smithi057.stdout:client_res_new = .400000 2024-01-10T06:50:09.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-01-10T06:50:10.618 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-01-10T06:50:11.059 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-01-10T06:50:11.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-01-10T06:50:11.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-01-10T06:50:11.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-01-10T06:50:11.063 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-01-10T06:50:11.063 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-01-10T06:50:11.063 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-01-10T06:50:11.064 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:50:11.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-01-10T06:50:11.169 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-01-10T06:50:11.169 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:11.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:11.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:11.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:11.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:11.278 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:11.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:11.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:11.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:11.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:11.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:11.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:11.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:11.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:11.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:11.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:11.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:11.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:11.291 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:11.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:11.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:11.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:11.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:11.293 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:11.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:11.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:11.296 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:11.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:11.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:11.298 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:11.299 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.299 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:11.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:11.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:11.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-01-10T06:50:11.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:11.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:11.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:11.302 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:11.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:11.303 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:11.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:11.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:11.304 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:11.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:11.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:11.306 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:11.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:11.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:11.309 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:11.309 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.309 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:11.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:11.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:11.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:50:11.311 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:50:11.311 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.311 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:50:11.313 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:11.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:11.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:11.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:11.315 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:11.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:11.316 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:11.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:11.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:11.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:11.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:11.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:11.319 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:11.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:11.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:11.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:11.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:11.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:11.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:11.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:11.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:50:11.324 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:50:11.324 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.324 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:50:11.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:50:11.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:50:11.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:50:11.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:50:11.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-01-10T06:50:11.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-01-10T06:50:11.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:50:11.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:50:11.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:50:11.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-01-10T06:50:11.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:11.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:50:11.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:50:11.867 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:50:11.867 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:50:11.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:50:11.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:50:11.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get fsid 2024-01-10T06:50:11.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:50:11.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:50:11.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:50:11.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:50:11.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:50:11.975 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:50:11.976 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get mon_host 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-01-10T06:50:12.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:50:12.984 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:50:12.984 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:50:12.984 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:50:12.984 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:50:12.984 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:12.985 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:12.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:12.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:50:12.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:50:13.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:50:13.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:50:13.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:50:13.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ebc88fe9-729d-4a50-a0fb-81495ba9e060 2024-01-10T06:50:13.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ebc88fe9-729d-4a50-a0fb-81495ba9e060' 2024-01-10T06:50:13.015 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 ebc88fe9-729d-4a50-a0fb-81495ba9e060 2024-01-10T06:50:13.015 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:50:13.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlPp5lS5WaARAAEiugXOMfCkUPSJ5x6RTbVA== 2024-01-10T06:50:13.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlPp5lS5WaARAAEiugXOMfCkUPSJ5x6RTbVA=="}' 2024-01-10T06:50:13.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ebc88fe9-729d-4a50-a0fb-81495ba9e060 -i td/mclock-config/0/new.json 2024-01-10T06:50:13.380 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:50:13.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-01-10T06:50:13.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAlPp5lS5WaARAAEiugXOMfCkUPSJ5x6RTbVA== --osd-uuid ebc88fe9-729d-4a50-a0fb-81495ba9e060 2024-01-10T06:50:13.416 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.416+0000 7f81aca0a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:13.461 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.460+0000 7f81aca0a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:13.461 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.461+0000 7f81aca0a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:13.462 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.462+0000 7f81aca0a540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:50:13.462 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.462+0000 7f81aca0a540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:50:13.462 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.462+0000 7f81aca0a540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:50:13.493 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:13.492+0000 7f81aca0a540 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-01-10T06:50:19.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-01-10T06:50:19.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:50:19.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:50:19.164 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:50:19.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:50:19.402 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:50:19.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:50:19.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T06:50:19.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:50:19.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:50:19.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:50:19.424 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:19.424+0000 7fc7261df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:19.484 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:19.483+0000 7fc7261df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:19.485 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:19.484+0000 7fc7261df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:50:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:50:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:50:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:50:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:50:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:19.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:50:19.875 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:50:19.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:19.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:20.016 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:20.016+0000 7fc7261df540 -1 Falling back to public interface 2024-01-10T06:50:20.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:21.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:21.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:21.317 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:50:21.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:50:21.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:21.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:21.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:22.565 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:22.565+0000 7fc7261df540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:50:22.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:22.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:22.786 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:50:22.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:50:22.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:22.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:23.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:24.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:24.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:24.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:50:24.305 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:50:24.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:24.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:24.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:25.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:25.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:25.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:50:25.766 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:50:25.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:25.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:26.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:27.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:27.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:27.349 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:50:27.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:50:27.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:27.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:27.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:28.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:28.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:28.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:50:28.808 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:50:28.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:28.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:29.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:30.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:30.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:30.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:50:30.416 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:50:30.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:30.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:30.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:31.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:31.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:31.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:50:31.874 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:50:31.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:31.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:32.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:33.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:33.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:33.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:50:33.498 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:50:33.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:33.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:33.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:34.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:34.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:34.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:50:34.971 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:50:34.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:34.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:35.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:36.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:36.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:36.860 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:50:36.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:50:36.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:36.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:37.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:38.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:38.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:38.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:50:38.755 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:50:38.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:38.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:39.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:40.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:40.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:40.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:50:40.733 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:50:40.734 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:40.734 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:42.674 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:42.674 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:42.674 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:50:42.674 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:50:42.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:42.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:43.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:44.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:44.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:44.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:50:44.247 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:50:44.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:44.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:45.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:50:46.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:50:46.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:50:46.168 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:50:46.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:50:46.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:50:46.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:50:46.777 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42647,v1:127.0.0.1:6803/42647] [v2:127.0.0.1:6804/42647,v1:127.0.0.1:6805/42647] exists,up ebc88fe9-729d-4a50-a0fb-81495ba9e060 2024-01-10T06:50:46.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:50:46.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:50:46.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:50:46.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-01-10T06:50:46.778 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-01-10T06:50:47.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-01-10T06:50:47.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:47.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:47.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:50:47.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-01-10T06:50:47.388 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-01-10T06:50:47.388 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-01-10T06:50:47.389 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-01-10T06:50:47.389 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-01-10T06:50:47.389 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-01-10T06:50:47.532 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:50:47.532 INFO:tasks.workunit.client.0.smithi057.stdout: "success": "" 2024-01-10T06:50:47.532 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:50:47.543 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:47.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:50:47.651 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-01-10T06:50:47.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-01-10T06:50:47.651 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-01-10T06:50:47.651 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-01-10T06:50:47.651 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-01-10T06:50:47.651 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:47.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:50:47.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-01-10T06:50:47.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-01-10T06:50:47.761 INFO:tasks.workunit.client.0.smithi057.stdout:Original client_res for osd.0 = 0.300000 2024-01-10T06:50:47.761 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-01-10T06:50:47.761 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:47.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-01-10T06:50:47.762 INFO:tasks.workunit.client.0.smithi057.stdout:client_res_new = .400000 2024-01-10T06:50:47.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-01-10T06:50:47.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-01-10T06:50:48.251 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-01-10T06:50:48.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-01-10T06:50:48.700 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-01-10T06:50:48.700 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:48.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-01-10T06:50:48.702 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-01-10T06:50:48.702 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-01-10T06:50:48.702 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:48.702 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:48.702 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:48.702 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:48.703 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:48.703 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:48.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:48.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:48.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:50:48.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-01-10T06:50:48.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-01-10T06:50:48.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-01-10T06:50:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-01-10T06:50:48.936 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:50:48.936 INFO:tasks.workunit.client.0.smithi057.stdout: "success": "" 2024-01-10T06:50:48.936 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:50:48.947 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:48.948 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:50:49.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-01-10T06:50:49.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-01-10T06:50:49.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-01-10T06:50:49.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-01-10T06:50:49.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-01-10T06:50:49.505 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-01-10T06:50:49.505 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:49.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:49.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:50:49.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-01-10T06:50:49.615 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-01-10T06:50:49.615 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:49.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-01-10T06:50:49.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-01-10T06:50:50.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-01-10T06:50:55.124 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-01-10T06:50:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-01-10T06:50:55.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-01-10T06:50:55.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:55.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:50:55.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-01-10T06:50:55.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:50:55.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-01-10T06:50:55.695 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-01-10T06:50:55.695 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-01-10T06:50:55.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-01-10T06:50:55.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:55.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:55.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:55.803 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:55.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:55.803 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:55.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:55.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:55.805 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:55.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:55.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:55.807 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:55.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:55.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:55.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:55.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:55.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:55.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:55.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:55.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:55.819 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:55.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:55.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:55.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:55.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:55.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:55.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:55.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:55.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:55.822 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:55.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:55.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:55.824 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:55.824 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.824 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.824 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:55.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:55.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:55.827 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:55.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:55.828 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:55.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:55.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:55.829 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:55.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:55.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:55.831 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:55.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:55.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:55.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:55.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:55.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:55.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:55.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:50:55.836 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:50:55.836 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.837 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:50:55.839 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:50:55.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:50:55.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:50:55.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:50:55.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-01-10T06:50:55.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:50:55.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:50:55.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:50:55.841 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:50:55.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:50:55.841 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:50:55.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:50:55.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:50:55.842 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:50:55.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:50:55.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:50:55.844 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:50:55.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:50:55.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:50:55.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:50:55.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:50:55.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:50:55.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:50:55.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:50:55.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:50:55.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:50:55.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:55.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:55.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:50:55.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:50:55.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:50:55.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-01-10T06:50:55.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:56.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:50:56.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:50:56.641 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:50:56.641 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:50:56.641 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:50:56.641 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:50:56.642 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:50:56.642 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:50:56.642 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:50:56.642 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:50:56.643 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:56.643 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:56.643 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:56.643 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:50:56.643 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:50:56.643 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get fsid 2024-01-10T06:50:56.756 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:50:56.756 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:56.757 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:56.758 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:50:56.758 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:50:56.758 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get mon_host 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-01-10T06:50:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:57.752 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:57.753 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:50:57.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:50:57.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:50:57.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-01-10T06:50:57.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:50:57.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=486cfaee-1da3-4688-9515-c66f14f26b2b 2024-01-10T06:50:57.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 486cfaee-1da3-4688-9515-c66f14f26b2b' 2024-01-10T06:50:57.783 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 486cfaee-1da3-4688-9515-c66f14f26b2b 2024-01-10T06:50:57.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:50:57.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRPp5lsxVhLxAAe739PJD7Z65PVKAFtpBSgw== 2024-01-10T06:50:57.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRPp5lsxVhLxAAe739PJD7Z65PVKAFtpBSgw=="}' 2024-01-10T06:50:57.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 486cfaee-1da3-4688-9515-c66f14f26b2b -i td/mclock-config/0/new.json 2024-01-10T06:50:58.283 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:50:58.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-01-10T06:50:58.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBRPp5lsxVhLxAAe739PJD7Z65PVKAFtpBSgw== --osd-uuid 486cfaee-1da3-4688-9515-c66f14f26b2b 2024-01-10T06:50:58.319 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.318+0000 7f523173a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:58.321 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.320+0000 7f523173a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:58.322 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.321+0000 7f523173a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:50:58.322 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.322+0000 7f523173a540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:50:58.322 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.322+0000 7f523173a540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:50:58.322 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.322+0000 7f523173a540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:50:58.432 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:50:58.432+0000 7f523173a540 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-01-10T06:51:04.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-01-10T06:51:04.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:51:04.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:51:04.034 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:51:04.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:51:04.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:51:04.721 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:04.684+0000 7f55bddae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:04.686+0000 7f55bddae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:04.722 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:04.687+0000 7f55bddae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:51:05.207 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:51:05.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:05.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:05.210 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:05.210+0000 7f55bddae540 -1 Falling back to public interface 2024-01-10T06:51:05.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:06.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:06.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:06.668 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:51:06.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:51:06.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:06.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:07.817 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:07.816+0000 7f55bddae540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:51:08.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:08.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:08.211 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:51:08.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:51:08.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:08.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:08.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:09.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:09.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:09.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:51:09.835 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:51:09.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:09.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:10.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:11.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:11.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:11.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:51:11.292 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:51:11.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:11.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:11.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:12.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:12.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:12.750 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:51:12.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:51:12.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:12.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:13.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:14.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:14.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:14.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:51:14.209 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:51:14.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:14.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:14.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:15.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:15.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:15.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:51:15.695 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:51:15.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:15.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:16.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:17.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:17.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:17.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:51:17.157 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:51:17.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:17.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:17.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:18.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:18.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:18.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:51:18.693 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:51:18.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:18.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:19.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:20.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:20.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:20.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:51:20.153 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:51:20.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:20.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:20.838 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:21.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:21.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:21.840 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:51:21.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:51:21.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:21.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:22.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:23.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:23.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:51:23.453 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:51:23.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:23.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:25.570 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:25.570 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:25.571 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:51:25.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:51:25.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:25.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:26.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:27.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:27.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:27.204 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:51:27.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:51:27.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:27.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:28.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:29.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:29.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:29.537 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:51:29.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:51:29.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:29.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:30.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:31.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:31.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:31.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:51:31.500 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:51:31.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:31.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:32.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:33.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:33.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:33.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:51:33.199 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:51:33.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:33.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:33.669 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/45000,v1:127.0.0.1:6803/45000] [v2:127.0.0.1:6804/45000,v1:127.0.0.1:6805/45000] exists,up 486cfaee-1da3-4688-9515-c66f14f26b2b 2024-01-10T06:51:33.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:51:33.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:51:33.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:51:33.670 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-01-10T06:51:34.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-01-10T06:51:34.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-01-10T06:51:34.197 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-01-10T06:51:34.197 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-01-10T06:51:34.197 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-01-10T06:51:34.198 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=("osd_mclock_scheduler_background_recovery_res" "osd_mclock_scheduler_client_res") 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-01-10T06:51:34.306 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-01-10T06:51:34.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-01-10T06:51:34.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-01-10T06:51:34.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-01-10T06:51:34.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:34.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-01-10T06:51:34.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-01-10T06:51:35.140 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-01-10T06:51:35.140 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-01-10T06:51:35.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-01-10T06:51:35.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-01-10T06:51:35.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-01-10T06:51:37.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-01-10T06:51:37.161 INFO:tasks.workunit.client.0.smithi057.stdout:Check configuration values - Attempt#: 0 2024-01-10T06:51:37.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-01-10T06:51:37.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-01-10T06:51:37.612 INFO:tasks.workunit.client.0.smithi057.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-01-10T06:51:37.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-01-10T06:51:37.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-01-10T06:51:37.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:37.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-01-10T06:51:37.613 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-01-10T06:51:37.614 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-01-10T06:51:37.614 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-01-10T06:51:37.614 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-01-10T06:51:38.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-01-10T06:51:38.073 INFO:tasks.workunit.client.0.smithi057.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-01-10T06:51:38.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-01-10T06:51:38.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-01-10T06:51:38.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:38.074 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-01-10T06:51:38.075 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:38.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-01-10T06:51:38.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-01-10T06:51:38.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-01-10T06:51:38.184 INFO:tasks.workunit.client.0.smithi057.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-01-10T06:51:38.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-01-10T06:51:38.185 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-01-10T06:51:38.185 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:38.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-01-10T06:51:38.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:51:38.189 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:38.190 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:38.190 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:38.190 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:51:38.190 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-01-10T06:51:38.190 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:51:38.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-01-10T06:51:38.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-01-10T06:51:38.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:38.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-01-10T06:51:38.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-01-10T06:51:38.936 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-01-10T06:51:38.937 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-01-10T06:51:38.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-01-10T06:51:38.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-01-10T06:51:38.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-01-10T06:51:40.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-01-10T06:51:40.939 INFO:tasks.workunit.client.0.smithi057.stdout:Check configuration values - Attempt#: 0 2024-01-10T06:51:40.939 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-01-10T06:51:41.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-01-10T06:51:41.455 INFO:tasks.workunit.client.0.smithi057.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-01-10T06:51:41.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-01-10T06:51:41.455 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-01-10T06:51:41.455 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:41.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-01-10T06:51:41.456 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-01-10T06:51:41.457 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-01-10T06:51:41.457 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-01-10T06:51:41.457 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-01-10T06:51:41.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-01-10T06:51:41.904 INFO:tasks.workunit.client.0.smithi057.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-01-10T06:51:41.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-01-10T06:51:41.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-01-10T06:51:41.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:41.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-01-10T06:51:41.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:41.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-01-10T06:51:41.908 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-01-10T06:51:41.908 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-01-10T06:51:41.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-01-10T06:51:41.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:51:41.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:51:41.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:41.909 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:41.909 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:41.909 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:51:41.909 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-01-10T06:51:41.909 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:51:42.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-01-10T06:51:42.016 INFO:tasks.workunit.client.0.smithi057.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-01-10T06:51:42.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-01-10T06:51:42.017 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-01-10T06:51:42.017 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:42.018 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-01-10T06:51:42.018 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:51:42.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:51:42.021 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:51:42.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:51:42.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:51:42.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:51:42.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:51:42.126 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:51:42.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:51:42.127 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:51:42.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:51:42.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:51:42.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:51:42.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:51:42.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:51:42.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:51:42.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:51:42.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:51:42.137 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:51:42.137 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.137 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:51:42.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:51:42.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:51:42.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:51:42.140 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:51:42.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:51:42.142 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:51:42.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:51:42.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:51:42.143 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:51:42.144 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.144 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:51:42.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:51:42.145 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:51:42.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:51:42.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:51:42.148 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:51:42.148 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.148 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:51:42.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:51:42.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:51:42.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:51:42.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:51:42.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:51:42.151 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:51:42.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:51:42.152 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:51:42.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:51:42.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:51:42.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:51:42.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:51:42.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:51:42.155 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:51:42.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:51:42.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:51:42.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:51:42.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:51:42.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:51:42.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:51:42.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:51:42.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:51:42.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:51:42.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:51:42.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:51:42.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:51:42.165 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:51:42.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:51:42.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:51:42.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:51:42.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:51:42.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:51:42.168 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:51:42.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:51:42.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:51:42.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:51:42.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:51:42.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:51:42.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:51:42.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:51:42.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:51:42.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:51:42.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:51:42.175 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:51:42.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:51:42.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:51:42.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-01-10T06:51:42.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:51:42.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:51:42.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:51:42.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:51:42.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:51:42.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:51:42.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.862 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.863 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:51:42.863 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:51:42.863 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get fsid 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:51:42.974 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:51:42.975 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:51:42.975 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:42.975 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:42.975 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:51:42.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:51:42.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get mon_host 2024-01-10T06:51:43.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-01-10T06:51:43.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-01-10T06:51:43.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:51:43.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:51:43.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:51:43.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-01-10T06:51:43.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:51:43.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:51:43.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:51:43.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:51:44.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-01-10T06:51:44.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:51:44.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:51:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc259993-8084-4e80-bbde-1dc594dbe13b 2024-01-10T06:51:44.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dc259993-8084-4e80-bbde-1dc594dbe13b' 2024-01-10T06:51:44.011 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 dc259993-8084-4e80-bbde-1dc594dbe13b 2024-01-10T06:51:44.011 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:51:44.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAPp5lbxxQARAAtlIa/fIw04cb6VoM/cFbsw== 2024-01-10T06:51:44.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAPp5lbxxQARAAtlIa/fIw04cb6VoM/cFbsw=="}' 2024-01-10T06:51:44.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc259993-8084-4e80-bbde-1dc594dbe13b -i td/mclock-config/0/new.json 2024-01-10T06:51:44.410 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:51:44.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-01-10T06:51:44.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCAPp5lbxxQARAAtlIa/fIw04cb6VoM/cFbsw== --osd-uuid dc259993-8084-4e80-bbde-1dc594dbe13b 2024-01-10T06:51:44.447 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.446+0000 7f1e7f4b8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:44.449 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.448+0000 7f1e7f4b8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:44.450 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.449+0000 7f1e7f4b8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:44.451 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.449+0000 7f1e7f4b8540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:51:44.451 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.449+0000 7f1e7f4b8540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:51:44.451 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.449+0000 7f1e7f4b8540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:51:44.539 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:44.537+0000 7f1e7f4b8540 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-01-10T06:51:50.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-01-10T06:51:50.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:51:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:51:50.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:51:50.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:51:50.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:51:50.433 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:51:50.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:51:50.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T06:51:50.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:51:50.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:51:50.456 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:50.455+0000 7f7edca7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:50.464 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:50.462+0000 7f7edca7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:50.464 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:50.463+0000 7f7edca7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:51:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:51:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:51:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:51:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:51:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:51:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:50.878 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:51:50.878 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:51:50.878 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:50.878 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:51.000 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:51.000+0000 7f7edca7a540 -1 Falling back to public interface 2024-01-10T06:51:51.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:52.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:52.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:52.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:51:52.340 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:51:52.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:52.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:52.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:53.604 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:51:53.604+0000 7f7edca7a540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:51:53.796 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:53.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:53.797 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:51:53.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:51:53.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:53.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:54.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:55.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:55.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:55.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:51:55.270 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:51:55.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:55.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:55.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:56.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:56.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:56.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:51:56.804 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:51:56.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:56.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:57.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:58.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:58.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:58.352 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:51:58.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:51:58.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:58.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:51:58.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:51:59.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:51:59.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:51:59.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:51:59.812 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:51:59.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:51:59.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:00.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:01.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:01.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:01.394 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:52:01.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:52:01.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:01.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:01.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:02.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:02.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:02.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:52:02.865 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:52:02.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:02.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:03.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:04.473 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:52:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:52:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:04.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:05.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:05.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:05.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:52:05.957 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:52:05.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:05.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:06.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:07.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:07.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:07.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:52:07.540 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:52:07.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:07.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:08.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:09.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:09.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:09.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:52:09.308 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:52:09.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:09.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:10.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:11.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:11.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:11.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:52:11.520 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:52:11.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:11.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:12.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:13.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:13.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:13.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:52:13.280 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:52:13.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:13.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:14.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:15.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:15.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:15.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:52:15.443 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:52:15.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:15.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:16.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:17.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:17.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:17.074 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:52:17.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:52:17.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:17.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:17.594 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/47365,v1:127.0.0.1:6803/47365] [v2:127.0.0.1:6804/47365,v1:127.0.0.1:6805/47365] exists,up dc259993-8084-4e80-bbde-1dc594dbe13b 2024-01-10T06:52:17.594 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:52:17.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:52:17.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:52:17.595 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-01-10T06:52:18.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-01-10T06:52:18.106 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-01-10T06:52:18.106 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-01-10T06:52:18.106 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-01-10T06:52:18.106 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:52:18.106 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:52:18.106 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:18.107 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:18.107 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:18.107 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:52:18.107 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-01-10T06:52:18.107 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_profile 2024-01-10T06:52:18.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-01-10T06:52:18.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-01-10T06:52:18.215 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-01-10T06:52:18.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=("osd_mclock_scheduler_background_recovery_res" "osd_mclock_scheduler_client_res") 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-01-10T06:52:18.216 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:52:18.217 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-01-10T06:52:18.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-01-10T06:52:18.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-01-10T06:52:18.321 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-01-10T06:52:18.321 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:18.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-01-10T06:52:18.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-01-10T06:52:18.447 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:52:18.448 INFO:tasks.workunit.client.0.smithi057.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '0.800000' " 2024-01-10T06:52:18.448 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:52:18.458 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-01-10T06:52:18.459 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-01-10T06:52:18.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-01-10T06:52:18.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-01-10T06:52:18.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-01-10T06:52:20.461 INFO:tasks.workunit.client.0.smithi057.stdout:Check configuration values - Attempt#: 0 2024-01-10T06:52:20.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-01-10T06:52:20.462 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-01-10T06:52:20.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-01-10T06:52:20.917 INFO:tasks.workunit.client.0.smithi057.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-01-10T06:52:20.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-01-10T06:52:20.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-01-10T06:52:20.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-01-10T06:52:20.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-01-10T06:52:20.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-01-10T06:52:20.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-01-10T06:52:20.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-01-10T06:52:21.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-01-10T06:52:21.387 INFO:tasks.workunit.client.0.smithi057.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-01-10T06:52:21.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-01-10T06:52:21.388 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-01-10T06:52:21.388 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:21.389 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-01-10T06:52:21.389 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:21.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-01-10T06:52:21.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-01-10T06:52:21.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-01-10T06:52:21.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-01-10T06:52:21.498 INFO:tasks.workunit.client.0.smithi057.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-01-10T06:52:21.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-01-10T06:52:21.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:21.500 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-01-10T06:52:21.500 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:21.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-01-10T06:52:21.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-01-10T06:52:21.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-01-10T06:52:21.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-01-10T06:52:21.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:52:21.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-01-10T06:52:21.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-01-10T06:52:21.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:21.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-01-10T06:52:21.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-01-10T06:52:21.736 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:52:21.736 INFO:tasks.workunit.client.0.smithi057.stdout: "success": "osd_mclock_scheduler_client_res = '0.400000' " 2024-01-10T06:52:21.736 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:52:21.747 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-01-10T06:52:21.748 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-01-10T06:52:21.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-01-10T06:52:21.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-01-10T06:52:21.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-01-10T06:52:23.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-01-10T06:52:23.751 INFO:tasks.workunit.client.0.smithi057.stdout:Check configuration values - Attempt#: 0 2024-01-10T06:52:23.751 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-01-10T06:52:24.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-01-10T06:52:24.211 INFO:tasks.workunit.client.0.smithi057.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-01-10T06:52:24.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-01-10T06:52:24.212 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-01-10T06:52:24.212 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:24.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-01-10T06:52:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-01-10T06:52:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-01-10T06:52:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-01-10T06:52:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-01-10T06:52:24.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-01-10T06:52:24.673 INFO:tasks.workunit.client.0.smithi057.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-01-10T06:52:24.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-01-10T06:52:24.673 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-01-10T06:52:24.673 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:24.674 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-01-10T06:52:24.674 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:24.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.677 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.680 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:52:24.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-01-10T06:52:24.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-01-10T06:52:24.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-01-10T06:52:24.784 INFO:tasks.workunit.client.0.smithi057.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-01-10T06:52:24.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-01-10T06:52:24.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-01-10T06:52:24.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:24.785 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-01-10T06:52:24.786 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:52:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:52:24.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:52:24.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:52:24.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:52:24.895 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:52:24.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:52:24.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:52:24.896 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:52:24.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:52:24.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:52:24.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:52:24.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:52:24.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:52:24.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:52:24.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:52:24.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:52:24.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:52:24.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:52:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:52:24.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:52:24.908 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:52:24.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:52:24.909 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:52:24.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:52:24.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:52:24.910 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:52:24.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:52:24.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:52:24.912 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:52:24.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:52:24.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:52:24.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:52:24.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:52:24.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:52:24.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:52:24.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:52:24.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:52:24.920 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:52:24.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:52:24.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:52:24.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:52:24.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:52:24.923 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:52:24.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:52:24.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:52:24.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:52:24.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:52:24.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:52:24.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:52:24.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:52:24.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:52:24.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:52:24.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:52:24.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:52:24.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:52:24.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:52:24.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-01-10T06:52:24.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:52:24.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:52:24.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:52:24.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:52:24.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:52:24.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:52:24.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:52:24.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:52:24.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:52:24.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:52:24.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:52:24.936 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:52:24.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:52:24.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:52:24.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:52:24.938 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:52:24.938 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.939 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:52:24.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:52:24.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:52:24.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-01-10T06:52:24.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:52:24.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:24.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:24.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36266 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:52:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-01-10T06:52:24.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:52:25.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:52:25.605 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:52:25.606 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:52:25.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:52:25.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:52:25.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:52:25.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:25.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:52:25.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:52:25.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get fsid 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:25.716 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:25.717 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-mon.a.asok 2024-01-10T06:52:25.717 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:52:25.717 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36266/ceph-mon.a.asok config get mon_host 2024-01-10T06:52:25.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-01-10T06:52:25.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-01-10T06:52:25.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:52:25.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:52:25.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:52:25.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-01-10T06:52:25.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:52:26.712 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:52:26.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:52:26.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-01-10T06:52:26.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:52:26.741 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:52:26.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T06:52:26.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-01-10T06:52:26.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:52:26.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f910ec1e-3a4c-4ca8-ac0d-4c93f58d51ff 2024-01-10T06:52:26.743 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 f910ec1e-3a4c-4ca8-ac0d-4c93f58d51ff 2024-01-10T06:52:26.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f910ec1e-3a4c-4ca8-ac0d-4c93f58d51ff' 2024-01-10T06:52:26.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:52:26.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqPp5lTBf0LBAAoNs7UQRFJa/vxjOd26yPCA== 2024-01-10T06:52:26.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqPp5lTBf0LBAAoNs7UQRFJa/vxjOd26yPCA=="}' 2024-01-10T06:52:26.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f910ec1e-3a4c-4ca8-ac0d-4c93f58d51ff -i td/mclock-config/0/new.json 2024-01-10T06:52:27.194 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:52:27.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-01-10T06:52:27.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCqPp5lTBf0LBAAoNs7UQRFJa/vxjOd26yPCA== --osd-uuid f910ec1e-3a4c-4ca8-ac0d-4c93f58d51ff 2024-01-10T06:52:27.231 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.230+0000 7f96a57b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:52:27.233 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.233+0000 7f96a57b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:52:27.234 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.234+0000 7f96a57b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:52:27.235 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.234+0000 7f96a57b0540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:52:27.235 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.234+0000 7f96a57b0540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:52:27.235 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.234+0000 7f96a57b0540 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-01-10T06:52:27.324 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:27.323+0000 7f96a57b0540 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-01-10T06:52:33.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-01-10T06:52:33.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:52:33.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:52:33.244 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:52:33.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:52:33.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:52:33.899 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:52:33.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=29bb5f3b-6b2f-4f8a-aec4-fea48eaeda44 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.36266/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T06:52:33.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:52:33.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:52:33.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:52:33.922 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:33.921+0000 7fa40ab93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:52:33.924 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:33.924+0000 7fa40ab93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:52:33.925 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:33.924+0000 7fa40ab93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:52:34.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:52:34.424 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:52:34.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:52:34.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:52:34.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:52:34.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:52:34.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:34.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:52:34.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:34.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:34.449 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:34.449+0000 7fa40ab93540 -1 Falling back to public interface 2024-01-10T06:52:34.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:35.967 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:35.967 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:35.967 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:52:35.968 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:52:35.968 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:35.968 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:36.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:37.126 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:52:37.126+0000 7fa40ab93540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:52:37.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:37.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:37.433 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:52:37.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:52:37.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:37.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:37.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:38.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:38.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:38.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:52:38.925 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:52:38.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:38.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:39.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:40.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:40.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:40.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:52:40.377 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:52:40.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:40.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:40.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:41.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:41.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:41.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:52:41.979 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:52:41.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:41.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:42.430 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:43.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:43.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:43.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:52:43.432 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:52:43.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:43.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:44.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:45.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:45.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:45.013 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:52:45.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:52:45.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:45.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:45.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:46.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:46.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:46.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:52:46.470 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:52:46.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:46.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:46.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:47.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:47.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:47.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:52:47.976 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:52:47.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:47.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:48.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:49.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:49.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:49.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:52:49.436 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:52:49.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:49.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:50.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:51.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:51.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:51.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:52:51.314 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:52:51.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:51.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:52.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:53.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:53.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:53.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:52:53.123 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:52:53.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:53.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:53.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:54.824 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:54.824 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:54.824 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:52:54.824 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:52:54.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:54.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:55.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:56.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:56.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:56.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:52:56.938 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:52:56.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:56.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:57.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:52:58.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:52:58.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:52:58.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:52:58.833 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:52:58.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:52:58.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:52:59.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:00.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:00.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:00.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:53:00.648 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:53:00.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:00.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:01.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:02.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:02.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:02.346 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:53:02.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:53:02.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:02.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:02.807 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/49712,v1:127.0.0.1:6803/49712] [v2:127.0.0.1:6804/49712,v1:127.0.0.1:6805/49712] exists,up f910ec1e-3a4c-4ca8-ac0d-4c93f58d51ff 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:53:02.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:53:02.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_recovery_max_active 2024-01-10T06:53:02.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-01-10T06:53:02.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-01-10T06:53:02.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-01-10T06:53:02.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-01-10T06:53:03.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-01-10T06:53:05.579 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:53:05.579 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:53:05.579 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:53:05.579 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:53:05.579 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:05.580 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:53:05.580 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:53:05.580 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:53:05.580 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_recovery_max_active 2024-01-10T06:53:05.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-01-10T06:53:05.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-01-10T06:53:05.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-01-10T06:53:06.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-01-10T06:53:06.790 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-01-10T06:53:08.793 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-01-10T06:53:08.793 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:53:08.793 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:53:08.793 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:53:08.793 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:08.794 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:53:08.794 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36266/ceph-osd.0.asok 2024-01-10T06:53:08.794 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-01-10T06:53:08.794 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.36266/ceph-osd.0.asok config get osd_recovery_max_active 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:53:08.913 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:53:08.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:53:08.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:53:08.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:53:09.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:53:09.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:53:09.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:53:09.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:53:09.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:53:09.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:53:09.021 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:53:09.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:53:09.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:53:09.023 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:53:09.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:53:09.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:53:09.030 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:53:09.030 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.030 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:53:09.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:53:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:53:09.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:53:09.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:53:09.034 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:53:09.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:53:09.035 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:53:09.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:53:09.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:53:09.036 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:53:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:53:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:53:09.038 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:53:09.040 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:53:09.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:53:09.044 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:53:09.045 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.045 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:53:09.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:53:09.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:53:09.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:53:09.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2024-01-10T06:53:09.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/mclock-config 0 2024-01-10T06:53:09.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-01-10T06:53:09.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-01-10T06:53:09.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-01-10T06:53:09.046 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:53:09.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:53:09.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:53:09.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:53:09.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:53:09.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:53:09.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:53:09.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:53:09.050 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:53:09.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:53:09.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:53:09.051 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:53:09.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:53:09.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:53:09.051 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:53:09.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-01-10T06:53:09.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-01-10T06:53:09.055 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:53:09.055 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36266 2024-01-10T06:53:09.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36266 2024-01-10T06:53:09.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:53:09.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:53:09.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2024-01-10T06:53:09.057 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-10T06:53:09.057 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-10T06:53:09.088 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-01-10T06:53:09.088 DEBUG:teuthology.orchestra.run.smithi057:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-01-10T06:53:09.153 INFO:tasks.workunit.client.0.smithi057.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-01-10T06:53:09.162 INFO:tasks.workunit.client.0.smithi057.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-01-10T06:53:09.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T06:53:09.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T06:53:09.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2024-01-10T06:53:09.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2024-01-10T06:53:09.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/network-ping 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none ' 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-01-10T06:53:09.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-01-10T06:53:09.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-01-10T06:53:09.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-01-10T06:53:09.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-01-10T06:53:09.165 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_network_ping_test2' 2024-01-10T06:53:09.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-01-10T06:53:09.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-01-10T06:53:09.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:53:09.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:53:09.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:53:09.167 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:53:09.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:53:09.167 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:53:09.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:53:09.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:53:09.168 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:53:09.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:53:09.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:53:09.170 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:53:09.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:53:09.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:53:09.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-01-10T06:53:09.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:53:09.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:09.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50845 2024-01-10T06:53:09.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:53:09.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:53:09.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-01-10T06:53:09.175 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:53:09.175 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.175 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:09.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.50845 2024-01-10T06:53:09.177 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:53:09.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:53:09.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-01-10T06:53:09.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:53:09.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:53:09.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:53:09.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:53:09.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:53:09.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:53:09.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:53:09.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mon.a.asok 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:53:09.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.50845/ceph-mon.a.asok config get fsid 2024-01-10T06:53:09.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:53:09.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:53:09.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:53:09.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:53:09.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mon.a.asok 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:53:09.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.50845/ceph-mon.a.asok config get mon_host 2024-01-10T06:53:10.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-01-10T06:53:10.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-01-10T06:53:10.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:53:10.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:53:10.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:53:10.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-01-10T06:53:10.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:10.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:53:10.963 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:53:10.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:53:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:53:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-01-10T06:53:10.993 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:53:10.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d74fb71-6ebb-4695-b606-d33b83b41c09 2024-01-10T06:53:10.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8d74fb71-6ebb-4695-b606-d33b83b41c09' 2024-01-10T06:53:10.994 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 8d74fb71-6ebb-4695-b606-d33b83b41c09 2024-01-10T06:53:10.994 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:53:11.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXPp5lAMlvABAAqy9KL/ThPGHxHkh6r8FKkw== 2024-01-10T06:53:11.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXPp5lAMlvABAAqy9KL/ThPGHxHkh6r8FKkw=="}' 2024-01-10T06:53:11.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d74fb71-6ebb-4695-b606-d33b83b41c09 -i td/network-ping/0/new.json 2024-01-10T06:53:11.403 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:53:11.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-01-10T06:53:11.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXPp5lAMlvABAAqy9KL/ThPGHxHkh6r8FKkw== --osd-uuid 8d74fb71-6ebb-4695-b606-d33b83b41c09 2024-01-10T06:53:11.441 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.440+0000 7fb456431540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:11.443 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.443+0000 7fb456431540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:11.444 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.444+0000 7fb456431540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:11.444 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.444+0000 7fb456431540 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-01-10T06:53:11.445 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.444+0000 7fb456431540 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-01-10T06:53:11.445 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.444+0000 7fb456431540 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-01-10T06:53:11.550 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:11.550+0000 7fb456431540 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-01-10T06:53:17.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-01-10T06:53:17.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:53:17.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:53:17.276 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:53:17.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:53:17.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:53:17.915 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:53:17.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:53:17.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:53:17.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:53:17.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:53:17.939 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:17.939+0000 7f980188e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:17.941 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:17.941+0000 7f980188e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:17.942 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:17.942+0000 7f980188e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:53:18.432 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:53:18.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:18.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:18.464 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:18.464+0000 7f980188e540 -1 Falling back to public interface 2024-01-10T06:53:18.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:19.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:19.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:19.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:53:19.908 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:53:19.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:19.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:20.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:21.064 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:21.063+0000 7f980188e540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:53:21.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:21.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:21.446 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:53:21.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:53:21.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:21.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:22.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:23.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:23.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:23.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:53:23.026 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:53:23.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:23.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:23.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:24.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:24.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:24.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:53:24.494 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:53:24.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:24.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:24.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:25.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:25.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:25.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:53:25.995 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:53:25.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:25.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:26.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:27.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:27.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:27.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:53:27.452 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:53:27.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:27.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:28.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:29.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:29.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:29.021 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:53:29.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:53:29.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:29.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:29.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:30.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:30.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:30.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:53:30.495 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:53:30.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:30.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:30.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:31.982 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:31.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:31.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:53:31.983 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:53:31.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:31.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:32.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:33.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:33.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:33.439 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:53:33.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:53:33.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:33.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:34.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:35.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:35.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:35.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:53:35.520 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:53:35.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:35.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:36.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:37.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:37.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:37.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:53:37.180 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:53:37.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:37.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:37.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:38.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:38.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:38.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:53:38.706 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:53:38.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:38.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:39.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:40.685 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:40.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:40.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:53:40.686 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:53:40.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:40.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:41.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:42.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:42.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:42.203 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:53:42.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:53:42.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:42.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:43.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:53:44.008 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:53:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:44.608 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:45.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:45.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:45.609 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:53:45.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:53:45.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:45.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:53:46.079 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/51977,v1:127.0.0.1:6803/51977] [v2:127.0.0.1:6804/51977,v1:127.0.0.1:6805/51977] exists,up 8d74fb71-6ebb-4695-b606-d33b83b41c09 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:53:46.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:53:46.084 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:53:46.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4287ab52-6bd7-44d6-9080-7dfb49f57e95 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4287ab52-6bd7-44d6-9080-7dfb49f57e95' 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 4287ab52-6bd7-44d6-9080-7dfb49f57e95 2024-01-10T06:53:46.086 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:53:46.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6Pp5lzcviBRAA36aDwo7WZUdRvVv5elKynA== 2024-01-10T06:53:46.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6Pp5lzcviBRAA36aDwo7WZUdRvVv5elKynA=="}' 2024-01-10T06:53:46.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4287ab52-6bd7-44d6-9080-7dfb49f57e95 -i td/network-ping/1/new.json 2024-01-10T06:53:46.661 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:53:46.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-01-10T06:53:46.674 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6Pp5lzcviBRAA36aDwo7WZUdRvVv5elKynA== --osd-uuid 4287ab52-6bd7-44d6-9080-7dfb49f57e95 2024-01-10T06:53:46.700 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.700+0000 7f6a0eb01540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:46.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.701+0000 7f6a0eb01540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:46.703 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.702+0000 7f6a0eb01540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:46.703 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.703+0000 7f6a0eb01540 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-01-10T06:53:46.703 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.703+0000 7f6a0eb01540 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-01-10T06:53:46.703 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.703+0000 7f6a0eb01540 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-01-10T06:53:46.776 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:46.776+0000 7f6a0eb01540 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-01-10T06:53:53.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-01-10T06:53:53.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:53:53.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T06:53:53.255 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T06:53:53.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:53:53.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T06:53:53.884 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T06:53:53.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:53:53.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:53:53.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:53:53.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:53:53.909 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:53.907+0000 7f05e433c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:53.932 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:53.931+0000 7f05e433c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:53.933 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:53.932+0000 7f05e433c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:53:54.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T06:53:54.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:53:54.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T06:53:54.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:53:54.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:53:54.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:54.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:53:54.399 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:53:54.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:54.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:53:54.469 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:54.468+0000 7f05e433c540 -1 Falling back to public interface 2024-01-10T06:53:54.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:55.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:55.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:55.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:53:55.850 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:53:55.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:55.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:53:56.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:56.813 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:53:56.811+0000 7f05e433c540 -1 osd.1 0 log_to_monitors true 2024-01-10T06:53:57.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:57.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:57.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:53:57.327 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:53:57.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:57.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:53:58.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:53:59.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:53:59.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:53:59.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:53:59.015 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:53:59.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:53:59.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:53:59.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:00.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:00.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:00.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:54:00.491 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:54:00.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:00.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:01.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:02.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:02.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:02.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:54:02.066 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:54:02.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:02.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:02.560 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:03.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:03.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:03.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:54:03.561 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:54:03.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:03.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:04.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:05.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:05.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:05.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:54:05.026 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:54:05.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:05.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:05.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:06.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:06.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:06.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:54:06.506 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:54:06.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:06.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:07.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:08.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:08.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:08.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:54:08.001 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:54:08.002 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:08.002 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:08.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:09.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:09.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:09.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:54:09.481 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:54:09.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:09.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:10.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:11.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:11.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:11.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:54:11.050 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:54:11.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:11.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:11.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:12.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:12.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:12.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:54:12.896 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:54:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:13.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:14.690 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:14.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:14.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:54:14.691 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:54:14.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:14.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:15.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:16.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:16.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:16.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:54:16.327 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:54:16.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:16.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:16.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:17.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:17.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:17.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:54:17.837 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:54:17.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:17.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:18.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:19.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:19.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:19.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:54:19.513 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:54:19.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:19.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:20.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:21.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:21.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:21.397 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:54:21.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:54:21.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:21.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:21.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:22.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:22.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:22.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T06:54:22.862 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T06:54:22.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:22.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:54:23.386 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53760,v1:127.0.0.1:6811/53760] [v2:127.0.0.1:6812/53760,v1:127.0.0.1:6813/53760] exists,up 4287ab52-6bd7-44d6-9080-7dfb49f57e95 2024-01-10T06:54:23.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:54:23.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:54:23.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:54:23.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-01-10T06:54:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf0e3817-27dc-43d0-8d2d-620791c7ce33 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bf0e3817-27dc-43d0-8d2d-620791c7ce33' 2024-01-10T06:54:23.393 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 bf0e3817-27dc-43d0-8d2d-620791c7ce33 2024-01-10T06:54:23.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:54:23.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfP55l1C45GBAASTEzkjVu2MPxM5ntpZyLFw== 2024-01-10T06:54:23.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfP55l1C45GBAASTEzkjVu2MPxM5ntpZyLFw=="}' 2024-01-10T06:54:23.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf0e3817-27dc-43d0-8d2d-620791c7ce33 -i td/network-ping/2/new.json 2024-01-10T06:54:23.995 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:54:24.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-01-10T06:54:24.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfP55l1C45GBAASTEzkjVu2MPxM5ntpZyLFw== --osd-uuid bf0e3817-27dc-43d0-8d2d-620791c7ce33 2024-01-10T06:54:24.034 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.033+0000 7fd49733a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:54:24.035 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.035+0000 7fd49733a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:54:24.036 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.036+0000 7fd49733a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:54:24.037 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.036+0000 7fd49733a540 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-01-10T06:54:24.037 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.036+0000 7fd49733a540 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-01-10T06:54:24.037 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.036+0000 7fd49733a540 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-01-10T06:54:24.127 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:24.126+0000 7fd49733a540 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-01-10T06:54:30.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-01-10T06:54:30.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:54:30.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T06:54:30.542 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T06:54:30.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:54:31.206 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T06:54:31.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T06:54:31.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:54:31.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:54:31.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:54:31.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:54:31.230 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:31.230+0000 7fd1754c9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:54:31.232 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:31.232+0000 7fd1754c9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:54:31.233 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:31.232+0000 7fd1754c9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:54:31.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:54:31.726 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:54:31.727 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:31.727 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:31.751 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:31.751+0000 7fd1754c9540 -1 Falling back to public interface 2024-01-10T06:54:32.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:33.186 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:54:33.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:54:33.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:33.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:33.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:34.477 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:54:34.477+0000 7fd1754c9540 -1 osd.2 0 log_to_monitors true 2024-01-10T06:54:34.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:34.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:34.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:54:34.680 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:54:34.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:34.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:35.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:36.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:36.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:36.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:54:36.239 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:54:36.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:36.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:36.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:37.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:37.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:37.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:54:37.784 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:54:37.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:37.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:38.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:39.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:39.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:39.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:54:39.301 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:54:39.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:39.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:39.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:40.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:40.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:40.769 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:54:40.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:54:40.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:40.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:41.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:42.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:42.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:42.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:54:42.354 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:54:42.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:42.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:42.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:43.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:43.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:43.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:54:43.954 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:54:43.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:43.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:44.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:45.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:45.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:45.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:54:45.417 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:54:45.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:45.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:45.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:46.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:46.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:46.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:54:46.912 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:54:46.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:46.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:47.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:48.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:48.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:48.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:54:48.537 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:54:48.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:48.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:49.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:50.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:50.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:50.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:54:50.371 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:54:50.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:50.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:51.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:52.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:52.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:52.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:54:52.071 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:54:52.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:52.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:53.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:54.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:54.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:54.029 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:54:54.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:54:54.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:54.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:54.601 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:55.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:55.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:55.603 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:54:55.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:54:55.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:55.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:56.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:57.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:57.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:57.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:54:57.486 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:54:57.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:57.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:57.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:54:58.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:54:58.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:54:58.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:54:58.978 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:54:58.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:54:58.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:54:59.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:55:00.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:55:00.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:55:00.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T06:55:00.810 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T06:55:00.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:55:00.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:55:01.295 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/55562,v1:127.0.0.1:6819/55562] [v2:127.0.0.1:6820/55562,v1:127.0.0.1:6821/55562] exists,up bf0e3817-27dc-43d0-8d2d-620791c7ce33 2024-01-10T06:55:01.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:55:01.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:55:01.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:55:01.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-01-10T06:55:06.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-01-10T06:55:06.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-01-10T06:55:07.258 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'foo' already exists 2024-01-10T06:55:07.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T06:55:08.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-01-10T06:55:08.475 INFO:tasks.workunit.client.0.smithi057.stdout:hints = 1 2024-01-10T06:55:08.475 INFO:tasks.workunit.client.0.smithi057.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-01-10T06:55:08.475 INFO:tasks.workunit.client.0.smithi057.stdout:Object prefix: benchmark_data_smithi057_56557 2024-01-10T06:55:08.475 INFO:tasks.workunit.client.0.smithi057.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-10T06:55:08.475 INFO:tasks.workunit.client.0.smithi057.stdout: 0 0 0 0 0 0 - 0 2024-01-10T06:55:09.475 INFO:tasks.workunit.client.0.smithi057.stdout: 1 16 64 48 0.187485 0.1875 0.285283 0.262739 2024-01-10T06:55:10.475 INFO:tasks.workunit.client.0.smithi057.stdout: 2 16 126 110 0.214821 0.242188 0.252654 0.274941 2024-01-10T06:55:11.475 INFO:tasks.workunit.client.0.smithi057.stdout: 3 16 176 160 0.208307 0.195312 0.275396 0.277882 2024-01-10T06:55:12.475 INFO:tasks.workunit.client.0.smithi057.stdout: 4 16 238 222 0.216769 0.242188 0.269056 0.278958 2024-01-10T06:55:13.475 INFO:tasks.workunit.client.0.smithi057.stdout: 5 16 288 272 0.212472 0.195312 0.297581 0.282594 2024-01-10T06:55:14.476 INFO:tasks.workunit.client.0.smithi057.stdout: 6 16 350 334 0.217419 0.242188 0.334885 0.284087 2024-01-10T06:55:15.476 INFO:tasks.workunit.client.0.smithi057.stdout: 7 16 400 384 0.214256 0.195312 0.288942 0.282822 2024-01-10T06:55:16.476 INFO:tasks.workunit.client.0.smithi057.stdout: 8 16 464 448 0.21872 0.25 0.251349 0.281492 2024-01-10T06:55:17.476 INFO:tasks.workunit.client.0.smithi057.stdout: 9 16 526 510 0.221324 0.242188 0.298271 0.280775 2024-01-10T06:55:18.476 INFO:tasks.workunit.client.0.smithi057.stdout: 10 16 560 544 0.212471 0.132812 0.284398 0.280963 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Total time run: 10.1557 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Total writes made: 560 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Write size: 4096 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Object size: 4096 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Bandwidth (MB/sec): 0.215397 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Stddev Bandwidth: 0.037702 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Max bandwidth (MB/sec): 0.25 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Min bandwidth (MB/sec): 0.132812 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Average IOPS: 55 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Stddev IOPS: 9.65171 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Max IOPS: 64 2024-01-10T06:55:19.477 INFO:tasks.workunit.client.0.smithi057.stdout:Min IOPS: 34 2024-01-10T06:55:19.478 INFO:tasks.workunit.client.0.smithi057.stdout:Average Latency(s): 0.286976 2024-01-10T06:55:19.478 INFO:tasks.workunit.client.0.smithi057.stdout:Stddev Latency(s): 0.0435723 2024-01-10T06:55:19.478 INFO:tasks.workunit.client.0.smithi057.stdout:Max latency(s): 0.492644 2024-01-10T06:55:19.478 INFO:tasks.workunit.client.0.smithi057.stdout:Min latency(s): 0.226818 2024-01-10T06:55:19.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-01-10T06:55:29.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-01-10T06:55:29.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T06:55:29.561 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T06:55:30.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T06:55:30.042 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T06:55:30.042 INFO:tasks.workunit.client.0.smithi057.stderr:2' 2024-01-10T06:55:30.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T06:55:30.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:55:30.042 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T06:55:30.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836507 2024-01-10T06:55:30.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836507 2024-01-10T06:55:30.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836507' 2024-01-10T06:55:30.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:55:30.194 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T06:55:30.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672987 2024-01-10T06:55:30.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672987 2024-01-10T06:55:30.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836507 1-42949672987' 2024-01-10T06:55:30.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:55:30.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T06:55:30.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509450 2024-01-10T06:55:30.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509450 2024-01-10T06:55:30.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836507 1-42949672987 2-64424509450' 2024-01-10T06:55:30.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:55:30.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836507 2024-01-10T06:55:30.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:55:30.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T06:55:30.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836507 2024-01-10T06:55:30.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:55:30.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836507 2024-01-10T06:55:30.471 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836507 2024-01-10T06:55:30.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-01-10T06:55:30.472 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:55:30.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836505 -lt 21474836507 2024-01-10T06:55:30.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T06:55:31.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T06:55:31.960 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:55:32.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836508 -lt 21474836507 2024-01-10T06:55:32.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:55:32.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672987 2024-01-10T06:55:32.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:55:32.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T06:55:32.512 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672987 2024-01-10T06:55:32.512 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:55:32.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672987 2024-01-10T06:55:32.513 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672987 2024-01-10T06:55:32.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-01-10T06:55:32.514 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T06:55:32.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672987 -lt 42949672987 2024-01-10T06:55:32.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:55:32.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509450 2024-01-10T06:55:32.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:55:32.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T06:55:32.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509450 2024-01-10T06:55:32.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:55:32.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509450 2024-01-10T06:55:32.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-01-10T06:55:32.960 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509450 2024-01-10T06:55:32.960 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T06:55:33.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509451 -lt 64424509450 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:55:33.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:33.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 2024-01-10T06:55:33.574 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:33.575 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 1000, 2024-01-10T06:55:33.575 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:55:33.575 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:33.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:33.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:33.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:33.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-01-10T06:55:33.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:33.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 2024-01-10T06:55:33.680 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:33.681 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 1000, 2024-01-10T06:55:33.681 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:55:33.681 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:33.690 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.690 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:33.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:33.693 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.693 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:33.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-01-10T06:55:33.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:33.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 0 2024-01-10T06:55:33.796 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:33.796 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 0, 2024-01-10T06:55:33.796 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [ 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:31 2024", 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.384, 2024-01-10T06:55:33.797 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.327, 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.353 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.547, 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.547, 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.547 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.547, 2024-01-10T06:55:33.798 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.547, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.547 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.259 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:33 2024", 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.287, 2024-01-10T06:55:33.799 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.379, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.324 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.461, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 2.148, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.148 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.461, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 2.148, 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.148 2024-01-10T06:55:33.800 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.244 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:33 2024", 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.323, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.371, 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.315 2024-01-10T06:55:33.801 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.506, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 2.092, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.092 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.506, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 2.092, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.092 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.220 2024-01-10T06:55:33.802 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:31 2024", 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.349, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.326, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.335 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.803 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.490, 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.700, 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.700 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.490, 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.700, 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.700 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.287 2024-01-10T06:55:33.804 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T06:55:33.805 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T06:55:33.805 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:33.805 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.806 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:33.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-01-10T06:55:33.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:33.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:33.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 0 2024-01-10T06:55:33.902 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:33.902 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 0, 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [ 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:25 2024", 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.379, 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.360, 2024-01-10T06:55:33.903 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.393 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.300, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.246, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.246 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.458, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.458, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.458 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.904 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.244 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:28 2024", 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.342, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.387, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.326 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.905 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.249, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.249, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.196 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.407, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 2.148, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.148 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.255 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.906 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:28 2024", 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.328, 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.374, 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.313 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.907 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.265, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.234, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.194 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.421, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 2.092, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.092 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.325 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.908 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:25 2024", 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.319, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.332, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.369 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.217, 2024-01-10T06:55:33.909 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.207, 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.207 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.401, 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.427, 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.427 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.204 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:26 2024", 2024-01-10T06:55:33.910 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.308, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.324, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.352 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.294, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.261, 2024-01-10T06:55:33.911 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.261 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.333, 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.460, 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.460 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.283 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:30 2024", 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.912 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.324, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.337, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.347 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.302, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.246, 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.246 2024-01-10T06:55:33.913 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.377, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.472, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.472 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.302 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:27 2024", 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:33.914 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.324, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.335, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.334 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.262, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.260, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.224 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.413, 2024-01-10T06:55:33.915 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.413, 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.491 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.287 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:26 2024", 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.916 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.293, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.325, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.335 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.258, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.234, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.234 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.322, 2024-01-10T06:55:33.917 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.700, 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.700 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.239 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:27 2024", 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.918 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.318, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.323, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.312 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.238, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.225, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.213 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.431, 2024-01-10T06:55:33.919 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.467, 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.467 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.230 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:30 2024", 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:33.920 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.309, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.320, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.315 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.270, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.261, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.261 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.359, 2024-01-10T06:55:33.921 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.452, 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.452 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.270 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:30 2024", 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.922 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.266, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.298, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.312 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.242, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.230, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.230 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.319, 2024-01-10T06:55:33.923 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.406, 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.406 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.256 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:30 2024", 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:33.924 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.284, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.304, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.298 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.232, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.232, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.232 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.336, 2024-01-10T06:55:33.925 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.424, 2024-01-10T06:55:33.926 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.424 2024-01-10T06:55:33.926 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:33.926 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.239 2024-01-10T06:55:33.926 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T06:55:33.926 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T06:55:33.926 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:33.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:33.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-01-10T06:55:33.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:33.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:33.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:33.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-01-10T06:55:53.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-01-10T06:55:53.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T06:55:53.920 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T06:55:54.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T06:55:54.376 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T06:55:54.376 INFO:tasks.workunit.client.0.smithi057.stderr:2' 2024-01-10T06:55:54.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T06:55:54.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:55:54.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T06:55:54.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836514 2024-01-10T06:55:54.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836514 2024-01-10T06:55:54.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514' 2024-01-10T06:55:54.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:55:54.572 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T06:55:54.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672994 2024-01-10T06:55:54.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672994 2024-01-10T06:55:54.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514 1-42949672994' 2024-01-10T06:55:54.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:55:54.709 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T06:55:54.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509457 2024-01-10T06:55:54.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509457 2024-01-10T06:55:54.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-64424509457' 2024-01-10T06:55:54.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:55:54.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836514 2024-01-10T06:55:54.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:55:54.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T06:55:54.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836514 2024-01-10T06:55:54.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:55:54.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836514 2024-01-10T06:55:54.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-01-10T06:55:54.850 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836514 2024-01-10T06:55:54.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:55:55.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836514 -lt 21474836514 2024-01-10T06:55:55.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:55:55.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672994 2024-01-10T06:55:55.293 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:55:55.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T06:55:55.294 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672994 2024-01-10T06:55:55.294 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:55:55.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672994 2024-01-10T06:55:55.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-01-10T06:55:55.295 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672994 2024-01-10T06:55:55.296 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T06:55:55.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672994 -lt 42949672994 2024-01-10T06:55:55.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:55:55.744 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509457 2024-01-10T06:55:55.744 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:55:55.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T06:55:55.745 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509457 2024-01-10T06:55:55.745 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:55:55.746 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509457 2024-01-10T06:55:55.746 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-01-10T06:55:55.747 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509457 2024-01-10T06:55:55.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T06:55:56.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509457 -lt 64424509457 2024-01-10T06:55:56.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:56.257 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:55:56.257 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:55:56.257 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:55:56.258 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:56.258 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:56.258 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:56.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:55:56.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:56.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 2024-01-10T06:55:56.356 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:56.356 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 1000, 2024-01-10T06:55:56.356 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:55:56.357 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:56.366 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.366 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:56.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:56.370 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.370 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:56.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-01-10T06:55:56.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:56.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 2024-01-10T06:55:56.463 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:56.463 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 1000, 2024-01-10T06:55:56.463 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:55:56.463 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:56.472 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.472 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:56.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:56.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:56.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:55:56.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:56.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 0 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 0, 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [ 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:53 2024", 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.578 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.357, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.311, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.324 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.414, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.506, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.092 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.414, 2024-01-10T06:55:56.579 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.506, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.092 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.298 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:53 2024", 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.356, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.308, 2024-01-10T06:55:56.580 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.331 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.463, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.463, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.148 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.463, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.463, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.148 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.225 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.581 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:56 2024", 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.326, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.326, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.339 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.485, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.485, 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.547 2024-01-10T06:55:56.582 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.485, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.485, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.547 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.253 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:56 2024", 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.312, 2024-01-10T06:55:56.583 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.310, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.325 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.498, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.498, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.700 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.498, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.498, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.700 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.241 2024-01-10T06:55:56.584 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T06:55:56.585 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T06:55:56.585 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:56.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:56.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-01-10T06:55:56.590 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.590 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:56.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:56.594 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:55:56.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:55:56.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:55:56.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:55:56.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:55:56.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:55:56.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:55:56.595 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:55:56.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:55:56.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 0 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 0, 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [ 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:50 2024", 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.430, 2024-01-10T06:55:56.684 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.340, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.323 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.372, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.218, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.218 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.491, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.491, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.491 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.223 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.685 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:50 2024", 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.426, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.364, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.353 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.325, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.237, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.237 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.686 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.498, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.514, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.514 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.249 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:50 2024", 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.409, 2024-01-10T06:55:56.687 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.346, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.316 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.337, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.232, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.232 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.510, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.510, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.510 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.209 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.688 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:50 2024", 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.404, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.351, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.328 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.311, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.235, 2024-01-10T06:55:56.689 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.235 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.441, 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.497, 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.497 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.231 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:50 2024", 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.690 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.365, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.327, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.366 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.270, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.226, 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.226 2024-01-10T06:55:56.691 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.419, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.452, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.458 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.462 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:50 2024", 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.692 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.357, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.308, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.343 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.336, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.204, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.204 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.394, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.417, 2024-01-10T06:55:56.693 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.427 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.442 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:53 2024", 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.357, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.311, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.324 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.694 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.213, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.213, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.202 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.414, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.506, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.092 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.239 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:53 2024", 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:55:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.356, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.308, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.331 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.249, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.216, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.216 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.463, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.463, 2024-01-10T06:55:56.696 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 2.148 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.256 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:52 2024", 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.348, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.299, 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.308 2024-01-10T06:55:56.697 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.304, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.227, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.213 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.389, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.434, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.467 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.401 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:51 2024", 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.698 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.347, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.337, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.342 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.286, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.252, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.252 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.477, 2024-01-10T06:55:56.699 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.547, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.547 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.285 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:51 2024", 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.345, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.318, 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.327 2024-01-10T06:55:56.700 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.243, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.239, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.234 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.491, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.491, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.700 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.701 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.270 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:55:52 2024", 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.342, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.307, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.321 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.266, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.235, 2024-01-10T06:55:56.702 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.224 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.408, 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.455, 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.470 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.420 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T06:55:56.703 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:55:56.704 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.705 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:55:56.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-01-10T06:55:56.705 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:55:56.705 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-01-10T06:55:56.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:55:56.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-01-10T06:56:46.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-01-10T06:56:46.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T06:56:46.701 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T06:56:47.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T06:56:47.169 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T06:56:47.169 INFO:tasks.workunit.client.0.smithi057.stderr:2' 2024-01-10T06:56:47.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T06:56:47.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:56:47.169 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T06:56:47.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836527 2024-01-10T06:56:47.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836527 2024-01-10T06:56:47.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836527' 2024-01-10T06:56:47.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:56:47.308 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T06:56:47.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949673006 2024-01-10T06:56:47.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949673006 2024-01-10T06:56:47.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836527 1-42949673006' 2024-01-10T06:56:47.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:56:47.444 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T06:56:47.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509470 2024-01-10T06:56:47.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509470 2024-01-10T06:56:47.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836527 1-42949673006 2-64424509470' 2024-01-10T06:56:47.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:56:47.582 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836527 2024-01-10T06:56:47.582 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:56:47.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T06:56:47.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836527 2024-01-10T06:56:47.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:56:47.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836527 2024-01-10T06:56:47.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-01-10T06:56:47.585 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836527 2024-01-10T06:56:47.585 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:56:48.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836525 -lt 21474836527 2024-01-10T06:56:48.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T06:56:49.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T06:56:49.063 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:56:49.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836527 -lt 21474836527 2024-01-10T06:56:49.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:56:49.580 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949673006 2024-01-10T06:56:49.580 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:56:49.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T06:56:49.581 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949673006 2024-01-10T06:56:49.581 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:56:49.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949673006 2024-01-10T06:56:49.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2024-01-10T06:56:49.583 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949673006 2024-01-10T06:56:49.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T06:56:50.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949673007 -lt 42949673006 2024-01-10T06:56:50.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:56:50.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509470 2024-01-10T06:56:50.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:56:50.028 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T06:56:50.029 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509470 2024-01-10T06:56:50.029 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:56:50.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509470 2024-01-10T06:56:50.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-01-10T06:56:50.030 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509470 2024-01-10T06:56:50.031 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T06:56:50.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509470 -lt 64424509470 2024-01-10T06:56:50.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:50.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:56:50.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:56:50.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 2024-01-10T06:56:50.581 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:56:50.581 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 1000, 2024-01-10T06:56:50.581 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:56:50.581 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:56:50.591 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.591 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:56:50.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:56:50.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-01-10T06:56:50.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-01-10T06:56:50.597 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:56:50.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 2024-01-10T06:56:50.687 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:56:50.687 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 1000, 2024-01-10T06:56:50.687 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:56:50.688 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:56:50.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:56:50.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:56:50.700 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.700 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-01-10T06:56:50.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-01-10T06:56:50.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:56:50.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:56:50.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 0 2024-01-10T06:56:50.803 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:56:50.803 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 0, 2024-01-10T06:56:50.803 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [ 2024-01-10T06:56:50.803 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:46 2024", 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.310, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.327, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.336 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.451, 2024-01-10T06:56:50.804 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.477, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.485 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.451, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.477, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.485 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.299 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:46 2024", 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.805 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.305, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.311, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.320 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.427, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.472, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.521 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.427, 2024-01-10T06:56:50.806 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.472, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.521 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.252 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:48 2024", 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.288, 2024-01-10T06:56:50.807 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.313, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.313 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.313, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.452, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.489 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.313, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.452, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.489 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.282 2024-01-10T06:56:50.808 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:48 2024", 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.277, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.299, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.305 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.306, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.441, 2024-01-10T06:56:50.809 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.505 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.306, 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.441, 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.505 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.270 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T06:56:50.810 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:56:50.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:56:50.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-01-10T06:56:50.816 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.816 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-01-10T06:56:50.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:56:50.819 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:56:50.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 0 2024-01-10T06:56:50.909 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 0, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [ 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:45 2024", 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.419, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.380, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.349 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.910 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.300, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.245, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.226 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.497, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.510, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.510 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.335 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:45 2024", 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.911 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.403, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.355, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.328 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.264, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.233, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.217 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.912 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.471, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.471, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.474 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.285 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:47 2024", 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.372, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.360, 2024-01-10T06:56:50.913 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.328 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.260, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.229, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.205 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.449, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.489, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.489 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.292 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.914 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:47 2024", 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 1, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.356, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.347, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.322 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.261, 2024-01-10T06:56:50.915 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.223, 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.216 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.426, 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.478, 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.511 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.261 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.916 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:45 2024", 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.323, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.352, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.352 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.255, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.255, 2024-01-10T06:56:50.917 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.231 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.404, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.518, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.518 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.278 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:45 2024", 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.918 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.327, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.343, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.339 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.243, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.243, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.209 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.418, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.464, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.510 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.261 2024-01-10T06:56:50.919 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:46 2024", 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.310, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.327, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.336 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.238, 2024-01-10T06:56:50.920 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.238, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.238 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.451, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.477, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.485 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.290 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:43 2024", 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.921 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.333, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.306, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.308 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.245, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.225, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.198 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.441, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.496, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.506 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.922 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.238 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:45 2024", 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:56:50.923 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.304, 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.325, 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.332 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.251, 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.251, 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.231 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.924 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.377, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.493, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.497 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.291 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:43 2024", 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 1, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "front", 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.331, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.318, 2024-01-10T06:56:50.925 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.313 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.238, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.238, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.216 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.452, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.469, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.489 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.257 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.926 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:46 2024", 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 0, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 2, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.305, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.311, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.320 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.262, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.223, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.223 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.927 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.427, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.472, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.521 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.272 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "last update": "Wed Jan 10 06:56:45 2024", 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "stale": false, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "from osd": 2, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "to osd": 0, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "interface": "back", 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "average": { 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.268, 2024-01-10T06:56:50.928 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.297, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.318 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "min": { 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.208, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.208, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.208 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "max": { 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "1min": 0.335, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "5min": 0.480, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "15min": 0.491 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: "last": 0.241 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T06:56:50.929 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-01-10T06:56:50.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mgr.x.asok 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:56:50.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-mgr.x.asok dump_osd_network 99 2024-01-10T06:56:51.015 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:56:51.015 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 99, 2024-01-10T06:56:51.015 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:56:51.015 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:56:51.024 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:51.024 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-01-10T06:56:51.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-01-10T06:56:51.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-osd.0.asok 2024-01-10T06:56:51.028 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-01-10T06:56:51.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.50845/ceph-osd.0.asok dump_osd_network 98 2024-01-10T06:56:51.127 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T06:56:51.127 INFO:tasks.workunit.client.0.smithi057.stdout: "threshold": 98, 2024-01-10T06:56:51.127 INFO:tasks.workunit.client.0.smithi057.stdout: "entries": [] 2024-01-10T06:56:51.128 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T06:56:51.137 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-01-10T06:56:51.137 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-01-10T06:56:51.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-01-10T06:56:51.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-01-10T06:56:51.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-01-10T06:56:51.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-01-10T06:56:51.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:56:51.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-01-10T06:56:51.141 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:56:51.141 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:56:51.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:56:51.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:56:51.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:56:51.249 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:56:51.250 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:56:51.250 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:56:51.250 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:56:51.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:56:51.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:56:51.252 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:56:51.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:56:51.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:56:51.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:56:51.254 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:56:51.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:56:51.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:56:51.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-01-10T06:56:51.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:56:51.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:51.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:51.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50845 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:56:51.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:56:51.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:56:51.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:56:51.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:56:51.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:56:51.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:56:51.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:56:51.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:56:51.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:56:51.278 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:56:51.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:56:51.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:56:51.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:56:51.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:56:51.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:56:51.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:56:51.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-01-10T06:56:51.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:56:51.284 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:51.284 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:51.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50845 2024-01-10T06:56:51.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:56:51.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:56:51.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-01-10T06:56:51.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:56:51.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:51.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:51.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.50845 2024-01-10T06:56:51.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:56:51.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T06:56:51.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:56:51.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-01-10T06:56:51.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-01-10T06:56:51.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-01-10T06:56:51.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:56:51.685 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T06:56:51.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:56:51.971 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:56:51.972 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:51.972 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:51.972 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:51.972 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mon.a.asok 2024-01-10T06:56:51.972 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:56:51.972 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.50845/ceph-mon.a.asok config get fsid 2024-01-10T06:56:52.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:56:52.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:56:52.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:56:52.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:56:52.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50845/ceph-mon.a.asok 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:56:52.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.50845/ceph-mon.a.asok config get mon_host 2024-01-10T06:56:52.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-01-10T06:56:52.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-01-10T06:56:52.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:56:52.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:56:52.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:56:52.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-01-10T06:56:52.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:56:53.088 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:56:53.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:56:53.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:56:53.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:56:53.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-01-10T06:56:53.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:56:53.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6a20fc9-897d-4bee-8c7f-773688b8505b 2024-01-10T06:56:53.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b6a20fc9-897d-4bee-8c7f-773688b8505b' 2024-01-10T06:56:53.122 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 b6a20fc9-897d-4bee-8c7f-773688b8505b 2024-01-10T06:56:53.122 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:56:53.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1P55lKRwZCBAA8t3n7M8jdM1Z7Et9ooZgcA== 2024-01-10T06:56:53.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1P55lKRwZCBAA8t3n7M8jdM1Z7Et9ooZgcA=="}' 2024-01-10T06:56:53.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6a20fc9-897d-4bee-8c7f-773688b8505b -i td/network-ping/0/new.json 2024-01-10T06:56:53.554 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:56:53.565 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-01-10T06:56:53.566 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1P55lKRwZCBAA8t3n7M8jdM1Z7Et9ooZgcA== --osd-uuid b6a20fc9-897d-4bee-8c7f-773688b8505b 2024-01-10T06:56:53.593 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.592+0000 7f44e987f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:56:53.595 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.595+0000 7f44e987f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:56:53.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.596+0000 7f44e987f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:56:53.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.596+0000 7f44e987f540 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-01-10T06:56:53.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.596+0000 7f44e987f540 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-01-10T06:56:53.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.596+0000 7f44e987f540 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-01-10T06:56:53.679 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:56:53.678+0000 7f44e987f540 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-01-10T06:56:59.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-01-10T06:56:59.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:56:59.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:56:59.637 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:56:59.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:57:00.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:57:00.261 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:57:00.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:57:00.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:57:00.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:57:00.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:57:00.286 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:00.285+0000 7f79283ba540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:00.287 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:00.287+0000 7f79283ba540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:00.288 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:00.288+0000 7f79283ba540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:00.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:57:00.725 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:57:00.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:00.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:00.827 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:00.827+0000 7f79283ba540 -1 Falling back to public interface 2024-01-10T06:57:01.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:02.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:02.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:02.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:57:02.193 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:57:02.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:02.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:02.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:03.355 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:03.355+0000 7f79283ba540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:57:03.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:03.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:03.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:57:03.669 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:57:03.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:03.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:04.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:05.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:05.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:05.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:57:05.157 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:57:05.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:05.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:05.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:06.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:06.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:06.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:57:06.614 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:57:06.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:06.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:07.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:08.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:08.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:08.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:57:08.184 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:57:08.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:08.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:08.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:09.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:09.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:09.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:57:09.681 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:57:09.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:09.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:10.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:11.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:11.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:11.321 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:57:11.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:57:11.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:11.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:11.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:12.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:12.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:12.810 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:57:12.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:57:12.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:12.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:13.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:14.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:14.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:14.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:57:14.281 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:57:14.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:14.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:14.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:15.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:15.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:15.770 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:57:15.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:57:15.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:15.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:16.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:17.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:17.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:17.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:57:17.228 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:57:17.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:17.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:18.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:19.522 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:19.522 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:19.522 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:57:19.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:57:19.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:19.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:19.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:20.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:20.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:20.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:57:20.981 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:57:20.982 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:20.982 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:21.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:22.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:22.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:22.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:57:22.843 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:57:22.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:22.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:23.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:24.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:24.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:24.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:57:24.311 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:57:24.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:24.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:24.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:25.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:25.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:25.994 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:57:25.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:57:25.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:25.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:26.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:27.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:27.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:27.642 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:57:27.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:57:27.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:27.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:28.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:29.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:29.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:29.233 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T06:57:29.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T06:57:29.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:29.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:57:29.697 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/58513,v1:127.0.0.1:6803/58513] [v2:127.0.0.1:6804/58513,v1:127.0.0.1:6805/58513] exists,up b6a20fc9-897d-4bee-8c7f-773688b8505b 2024-01-10T06:57:29.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:57:29.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:57:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:57:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=22217bde-265a-4484-adbe-8a1ac13d1389 2024-01-10T06:57:29.705 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 22217bde-265a-4484-adbe-8a1ac13d1389 2024-01-10T06:57:29.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 22217bde-265a-4484-adbe-8a1ac13d1389' 2024-01-10T06:57:29.705 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:57:29.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZP55lEjG+KhAADVMooXiWHtk0r1f1f+5qVg== 2024-01-10T06:57:29.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZP55lEjG+KhAADVMooXiWHtk0r1f1f+5qVg=="}' 2024-01-10T06:57:29.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 22217bde-265a-4484-adbe-8a1ac13d1389 -i td/network-ping/1/new.json 2024-01-10T06:57:30.287 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:57:30.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-01-10T06:57:30.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZP55lEjG+KhAADVMooXiWHtk0r1f1f+5qVg== --osd-uuid 22217bde-265a-4484-adbe-8a1ac13d1389 2024-01-10T06:57:30.326 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.326+0000 7f0832cfb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:30.328 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.328+0000 7f0832cfb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:30.329 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.329+0000 7f0832cfb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:30.330 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.329+0000 7f0832cfb540 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-01-10T06:57:30.330 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.329+0000 7f0832cfb540 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-01-10T06:57:30.330 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.329+0000 7f0832cfb540 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-01-10T06:57:30.377 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:30.377+0000 7f0832cfb540 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-01-10T06:57:36.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-01-10T06:57:36.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:57:36.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T06:57:36.993 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T06:57:36.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:57:37.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T06:57:37.626 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T06:57:37.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:57:37.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:57:37.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:57:37.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:57:37.650 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:37.650+0000 7fee04637540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:37.694 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:37.694+0000 7fee04637540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:37.695 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:37.695+0000 7fee04637540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:57:38.076 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:57:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:38.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:38.215 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:38.215+0000 7fee04637540 -1 Falling back to public interface 2024-01-10T06:57:38.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:57:39.552 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:57:39.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:39.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:40.880 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:57:40.879+0000 7fee04637540 -1 osd.1 0 log_to_monitors true 2024-01-10T06:57:41.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:41.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:41.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:57:41.027 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:57:41.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:41.028 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:41.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:42.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:42.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:42.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:57:42.666 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:57:42.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:42.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:43.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:44.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:44.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:44.177 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:57:44.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:57:44.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:44.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:44.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:45.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:45.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:45.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:57:45.771 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:57:45.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:45.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:46.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:47.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:47.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:47.239 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:57:47.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:57:47.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:47.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:47.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:48.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:48.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:48.857 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:57:48.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:57:48.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:48.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:49.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:50.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:50.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:50.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:57:50.364 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:57:50.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:50.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:50.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:51.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:51.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:51.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:57:51.910 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:57:51.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:51.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:52.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:53.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:53.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:53.476 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:57:53.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:57:53.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:53.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:53.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:54.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:54.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:54.955 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:57:54.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:57:54.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:54.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:55.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:56.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:56.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:56.420 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:57:56.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:57:56.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:56.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:57.727 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:57:58.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:57:58.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:57:58.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:57:58.729 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:57:58.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:57:58.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:57:59.445 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:00.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:00.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:00.448 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:58:00.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:58:00.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:00.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:58:01.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:02.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:02.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:02.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:58:02.241 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:58:02.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:02.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:58:03.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:04.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:04.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:04.220 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:58:04.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:58:04.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:04.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:58:05.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:06.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:06.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:06.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:58:06.375 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:58:06.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:06.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T06:58:06.881 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/60314,v1:127.0.0.1:6811/60314] [v2:127.0.0.1:6812/60314,v1:127.0.0.1:6813/60314] exists,up 22217bde-265a-4484-adbe-8a1ac13d1389 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:58:06.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:58:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:58:06.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=657e0ad9-210b-456d-b6f9-16c9c9411ae2 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 657e0ad9-210b-456d-b6f9-16c9c9411ae2' 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 657e0ad9-210b-456d-b6f9-16c9c9411ae2 2024-01-10T06:58:06.888 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:58:06.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+P55lF3mvNRAAHlEwwr997kcZrtqRX3iY5A== 2024-01-10T06:58:06.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+P55lF3mvNRAAHlEwwr997kcZrtqRX3iY5A=="}' 2024-01-10T06:58:06.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 657e0ad9-210b-456d-b6f9-16c9c9411ae2 -i td/network-ping/2/new.json 2024-01-10T06:58:07.650 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:58:07.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-01-10T06:58:07.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+P55lF3mvNRAAHlEwwr997kcZrtqRX3iY5A== --osd-uuid 657e0ad9-210b-456d-b6f9-16c9c9411ae2 2024-01-10T06:58:07.690 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.690+0000 7f4adec67540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:58:07.701 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.700+0000 7f4adec67540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:58:07.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.702+0000 7f4adec67540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:58:07.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.702+0000 7f4adec67540 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-01-10T06:58:07.703 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.702+0000 7f4adec67540 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-01-10T06:58:07.703 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.702+0000 7f4adec67540 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-01-10T06:58:07.826 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:07.826+0000 7f4adec67540 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-01-10T06:58:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-01-10T06:58:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:58:14.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T06:58:14.415 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T06:58:14.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:58:15.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T06:58:15.039 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T06:58:15.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6e85ebc0-c9e8-462a-a02c-0bace2b6b667 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.50845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:58:15.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:58:15.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:58:15.040 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:58:15.063 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:15.063+0000 7fd826ac5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:58:15.065 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:15.065+0000 7fd826ac5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:58:15.066 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:15.066+0000 7fd826ac5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:58:15.530 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:58:15.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:15.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:15.590 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:15.590+0000 7fd826ac5540 -1 Falling back to public interface 2024-01-10T06:58:16.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:17.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:17.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:17.018 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:58:17.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:58:17.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:17.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:17.525 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:18.165 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:58:18.165+0000 7fd826ac5540 -1 osd.2 0 log_to_monitors true 2024-01-10T06:58:18.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:18.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:18.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:58:18.527 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:58:18.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:18.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:19.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:20.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:20.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:20.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:58:20.050 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:58:20.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:20.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:20.594 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:21.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:21.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:21.597 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:58:21.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:58:21.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:21.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:22.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:23.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:23.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:23.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:58:23.170 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:58:23.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:23.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:23.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:24.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:24.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:24.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:58:24.726 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:58:24.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:24.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:25.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:26.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:26.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:26.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:58:26.190 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:58:26.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:26.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:26.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:27.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:27.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:27.648 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:58:27.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:58:27.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:27.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:28.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:29.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:29.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:29.283 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:58:29.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:58:29.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:29.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:29.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:30.761 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:30.761 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:30.761 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:58:30.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:58:30.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:30.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:31.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:32.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:32.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:32.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:58:32.224 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:58:32.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:32.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:32.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:33.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:33.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:33.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T06:58:33.827 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T06:58:33.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:33.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:34.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:35.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:35.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:35.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T06:58:35.454 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T06:58:35.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:35.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:36.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:37.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:37.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:37.384 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T06:58:37.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T06:58:37.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:37.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:38.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:39.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:39.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:39.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T06:58:39.203 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T06:58:39.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:39.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:39.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:40.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:40.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:40.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T06:58:40.904 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T06:58:40.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:40.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:42.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:43.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:43.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:43.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T06:58:43.068 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T06:58:43.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:43.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:43.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:58:44.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:58:44.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:58:44.730 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T06:58:44.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T06:58:44.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:58:44.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T06:58:45.280 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/62097,v1:127.0.0.1:6819/62097] [v2:127.0.0.1:6820/62097,v1:127.0.0.1:6821/62097] exists,up 657e0ad9-210b-456d-b6f9-16c9c9411ae2 2024-01-10T06:58:45.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T06:58:45.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T06:58:45.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T06:58:45.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-01-10T06:58:50.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-01-10T06:58:50.879 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'dc1' already exists 2024-01-10T06:58:50.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-01-10T06:58:51.520 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'dc2' already exists 2024-01-10T06:58:51.529 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-01-10T06:58:52.227 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'dc3' already exists 2024-01-10T06:58:52.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-01-10T06:58:52.875 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'rack1' already exists 2024-01-10T06:58:52.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-01-10T06:58:53.586 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'rack2' already exists 2024-01-10T06:58:53.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-01-10T06:58:54.296 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'rack3' already exists 2024-01-10T06:58:54.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-01-10T06:58:54.897 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'host1' already exists 2024-01-10T06:58:54.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-01-10T06:58:55.568 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'host2' already exists 2024-01-10T06:58:55.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-01-10T06:58:56.337 INFO:tasks.workunit.client.0.smithi057.stderr:bucket 'host3' already exists 2024-01-10T06:58:56.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-01-10T06:58:56.998 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-01-10T06:58:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-01-10T06:58:57.711 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-01-10T06:58:57.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-01-10T06:58:58.337 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-01-10T06:58:58.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-01-10T06:58:58.946 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-01-10T06:58:58.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-01-10T06:58:59.533 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-01-10T06:58:59.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-01-10T06:59:00.195 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-01-10T06:59:00.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-01-10T06:59:00.836 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-01-10T06:59:00.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-01-10T06:59:01.512 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-01-10T06:59:01.522 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-01-10T06:59:02.217 INFO:tasks.workunit.client.0.smithi057.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-01-10T06:59:02.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-01-10T06:59:02.902 INFO:tasks.workunit.client.0.smithi057.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-01-10T06:59:02.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-01-10T06:59:03.599 INFO:tasks.workunit.client.0.smithi057.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-01-10T06:59:03.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-01-10T06:59:04.316 INFO:tasks.workunit.client.0.smithi057.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-01-10T06:59:04.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-01-10T06:59:05.010 INFO:tasks.workunit.client.0.smithi057.stderr:rule myrule already exists 2024-01-10T06:59:05.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-01-10T06:59:05.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-01-10T06:59:05.686 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'foo' already exists 2024-01-10T06:59:05.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T06:59:06.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-01-10T06:59:06.820 INFO:tasks.workunit.client.0.smithi057.stdout:hints = 1 2024-01-10T06:59:06.820 INFO:tasks.workunit.client.0.smithi057.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-01-10T06:59:06.821 INFO:tasks.workunit.client.0.smithi057.stdout:Object prefix: benchmark_data_smithi057_63619 2024-01-10T06:59:06.821 INFO:tasks.workunit.client.0.smithi057.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-10T06:59:06.821 INFO:tasks.workunit.client.0.smithi057.stdout: 0 0 0 0 0 0 - 0 2024-01-10T06:59:07.821 INFO:tasks.workunit.client.0.smithi057.stdout: 1 16 60 44 0.171855 0.171875 0.270075 0.326979 2024-01-10T06:59:08.821 INFO:tasks.workunit.client.0.smithi057.stdout: 2 16 120 104 0.203096 0.234375 0.242782 0.294716 2024-01-10T06:59:09.821 INFO:tasks.workunit.client.0.smithi057.stdout: 3 16 174 158 0.205699 0.210938 0.301053 0.281005 2024-01-10T06:59:10.821 INFO:tasks.workunit.client.0.smithi057.stdout: 4 16 232 216 0.210906 0.226562 0.259955 0.285843 2024-01-10T06:59:11.821 INFO:tasks.workunit.client.0.smithi057.stdout: 5 16 296 280 0.218718 0.25 0.311615 0.280892 2024-01-10T06:59:12.821 INFO:tasks.workunit.client.0.smithi057.stdout: 6 16 345 329 0.214161 0.191406 0.325383 0.280462 2024-01-10T06:59:13.821 INFO:tasks.workunit.client.0.smithi057.stdout: 7 16 409 393 0.219275 0.25 0.275293 0.276206 2024-01-10T06:59:14.822 INFO:tasks.workunit.client.0.smithi057.stdout: 8 16 472 456 0.222623 0.246094 0.25279 0.276292 2024-01-10T06:59:15.822 INFO:tasks.workunit.client.0.smithi057.stdout: 9 16 526 510 0.221321 0.210938 0.266617 0.276513 2024-01-10T06:59:16.822 INFO:tasks.workunit.client.0.smithi057.stdout: 10 16 584 568 0.221841 0.226562 0.310224 0.278566 2024-01-10T06:59:17.822 INFO:tasks.workunit.client.0.smithi057.stdout:Total time run: 10.1801 2024-01-10T06:59:17.822 INFO:tasks.workunit.client.0.smithi057.stdout:Total writes made: 584 2024-01-10T06:59:17.822 INFO:tasks.workunit.client.0.smithi057.stdout:Write size: 4096 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Object size: 4096 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Bandwidth (MB/sec): 0.22409 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Stddev Bandwidth: 0.025898 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Max bandwidth (MB/sec): 0.25 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Min bandwidth (MB/sec): 0.171875 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Average IOPS: 57 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Stddev IOPS: 6.6299 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Max IOPS: 64 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Min IOPS: 44 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Average Latency(s): 0.278378 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Stddev Latency(s): 0.0475874 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Max latency(s): 0.587577 2024-01-10T06:59:17.823 INFO:tasks.workunit.client.0.smithi057.stdout:Min latency(s): 0.136868 2024-01-10T06:59:17.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-01-10T06:59:27.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-01-10T06:59:27.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T06:59:27.923 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T06:59:28.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T06:59:28.380 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T06:59:28.381 INFO:tasks.workunit.client.0.smithi057.stderr:2' 2024-01-10T06:59:28.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T06:59:28.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:59:28.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T06:59:28.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836509 2024-01-10T06:59:28.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836509 2024-01-10T06:59:28.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836509' 2024-01-10T06:59:28.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:59:28.543 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T06:59:28.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672983 2024-01-10T06:59:28.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672983 2024-01-10T06:59:28.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836509 1-42949672983' 2024-01-10T06:59:28.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T06:59:28.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T06:59:28.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509455 2024-01-10T06:59:28.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509455 2024-01-10T06:59:28.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836509 1-42949672983 2-64424509455' 2024-01-10T06:59:28.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:59:28.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836509 2024-01-10T06:59:28.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:59:28.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T06:59:28.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836509 2024-01-10T06:59:28.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:59:28.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836509 2024-01-10T06:59:28.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-01-10T06:59:28.820 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836509 2024-01-10T06:59:28.820 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:59:29.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836507 -lt 21474836509 2024-01-10T06:59:29.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T06:59:30.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T06:59:30.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T06:59:30.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836509 -lt 21474836509 2024-01-10T06:59:30.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:59:30.804 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672983 2024-01-10T06:59:30.804 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:59:30.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T06:59:30.805 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672983 2024-01-10T06:59:30.805 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:59:30.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672983 2024-01-10T06:59:30.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-01-10T06:59:30.806 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672983 2024-01-10T06:59:30.807 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T06:59:31.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672983 -lt 42949672983 2024-01-10T06:59:31.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T06:59:31.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509455 2024-01-10T06:59:31.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T06:59:31.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T06:59:31.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509455 2024-01-10T06:59:31.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T06:59:31.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509455 2024-01-10T06:59:31.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-01-10T06:59:31.272 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509455 2024-01-10T06:59:31.272 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T06:59:31.776 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509456 -lt 64424509455 2024-01-10T06:59:31.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-01-10T06:59:31.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-01-10T06:59:32.307 INFO:tasks.workunit.client.0.smithi057.stdout:HEALTH_ERR 2 mgr modules have failed; Slow OSD heartbeats on back (longest 0.395ms); Slow OSD heartbeats on front (longest 0.405ms); 1 pool(s) do not have an application enabled 2024-01-10T06:59:32.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-01-10T06:59:32.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-01-10T06:59:32.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-01-10T06:59:32.917 INFO:tasks.workunit.client.0.smithi057.stdout:HEALTH_ERR 2 mgr modules have failed; Slow OSD heartbeats on back (longest 0.395ms); Slow OSD heartbeats on front (longest 0.405ms); 1 pool(s) do not have an application enabled 2024-01-10T06:59:32.917 INFO:tasks.workunit.client.0.smithi057.stdout:[ERR] MGR_MODULE_ERROR: 2 mgr modules have failed 2024-01-10T06:59:32.917 INFO:tasks.workunit.client.0.smithi057.stdout: Module 'rbd_support' has failed: Not found or unloadable 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Module 'volumes' has failed: Not found or unloadable 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.395ms) 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.395 msec possibly improving 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.339 msec possibly improving 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.326 msec possibly improving 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.314 msec 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.308 msec possibly improving 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.299 msec 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.405ms) 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.405 msec possibly improving 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.346 msec 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.344 msec possibly improving 2024-01-10T06:59:32.918 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.340 msec possibly improving 2024-01-10T06:59:32.919 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.323 msec 2024-01-10T06:59:32.919 INFO:tasks.workunit.client.0.smithi057.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.312 msec 2024-01-10T06:59:32.919 INFO:tasks.workunit.client.0.smithi057.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-01-10T06:59:32.919 INFO:tasks.workunit.client.0.smithi057.stdout: application not enabled on pool 'foo' 2024-01-10T06:59:32.919 INFO:tasks.workunit.client.0.smithi057.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-01-10T06:59:32.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-01-10T06:59:32.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-01-10T06:59:32.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-01-10T06:59:32.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:59:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:59:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:59:33.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:59:33.044 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:59:33.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:59:33.044 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:59:33.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:59:33.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:59:33.045 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:59:33.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:59:33.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:59:33.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:59:33.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:59:33.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:59:33.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:59:33.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-01-10T06:59:33.066 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:59:33.066 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.067 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:59:33.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50845 2024-01-10T06:59:33.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:59:33.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:59:33.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2024-01-10T06:59:33.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/network-ping 0 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:59:33.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:59:33.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:59:33.070 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:59:33.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:59:33.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:59:33.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:59:33.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:59:33.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:59:33.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:59:33.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:59:33.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:59:33.074 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:59:33.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:59:33.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-01-10T06:59:33.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-01-10T06:59:33.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:59:33.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50845 2024-01-10T06:59:33.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50845 2024-01-10T06:59:33.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:59:33.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:59:33.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2024-01-10T06:59:33.079 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-10T06:59:33.079 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-10T06:59:33.107 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-01-10T06:59:33.107 DEBUG:teuthology.orchestra.run.smithi057:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-01-10T06:59:33.171 INFO:tasks.workunit.client.0.smithi057.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-01-10T06:59:33.181 INFO:tasks.workunit.client.0.smithi057.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-01-10T06:59:33.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T06:59:33.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/ok-to-stop 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-01-10T06:59:33.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none ' 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none ' 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none ' 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-01-10T06:59:33.183 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_0_osd 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_1_mon_checks 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_2_mons_checks 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_3_mons_checks 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_4_mons_checks 2024-01-10T06:59:33.185 INFO:tasks.workunit.client.0.smithi057.stderr:TEST_5_mons_checks' 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T06:59:33.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T06:59:33.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T06:59:33.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T06:59:33.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T06:59:33.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T06:59:33.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T06:59:33.188 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:59:33.188 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T06:59:33.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T06:59:33.189 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T06:59:33.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T06:59:33.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T06:59:33.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T06:59:33.191 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T06:59:33.191 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.191 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:33.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T06:59:33.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T06:59:33.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T06:59:33.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T06:59:33.193 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T06:59:33.193 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.193 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:33.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T06:59:33.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T06:59:33.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T06:59:33.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T06:59:33.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T06:59:33.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T06:59:33.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T06:59:33.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T06:59:33.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T06:59:33.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T06:59:33.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:59:33.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:59:33.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T06:59:33.869 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T06:59:33.869 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:59:33.870 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:59:33.870 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:59:33.870 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:59:33.871 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.871 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:33.871 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T06:59:33.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:59:33.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T06:59:33.980 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T06:59:33.980 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T06:59:33.980 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T06:59:33.980 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T06:59:33.981 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T06:59:34.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-01-10T06:59:34.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-01-10T06:59:34.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T06:59:34.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T06:59:34.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T06:59:34.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-01-10T06:59:34.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T06:59:34.977 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:59:34.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-01-10T06:59:35.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T06:59:35.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f83cefb6-c09f-4461-84b4-4e039534954b 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f83cefb6-c09f-4461-84b4-4e039534954b' 2024-01-10T06:59:35.007 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 f83cefb6-c09f-4461-84b4-4e039534954b 2024-01-10T06:59:35.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T06:59:35.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXQJ5lJ3cdARAAGt9IjnmcZSO28QiCMRm+qg== 2024-01-10T06:59:35.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXQJ5lJ3cdARAAGt9IjnmcZSO28QiCMRm+qg=="}' 2024-01-10T06:59:35.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f83cefb6-c09f-4461-84b4-4e039534954b -i td/ok-to-stop/0/new.json 2024-01-10T06:59:35.418 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:59:35.430 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-01-10T06:59:35.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXQJ5lJ3cdARAAGt9IjnmcZSO28QiCMRm+qg== --osd-uuid f83cefb6-c09f-4461-84b4-4e039534954b 2024-01-10T06:59:35.455 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.453+0000 7fa239f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:59:35.457 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.455+0000 7fa239f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:59:35.457 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.456+0000 7fa239f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:59:35.458 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.457+0000 7fa239f12540 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-01-10T06:59:35.458 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.457+0000 7fa239f12540 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-01-10T06:59:35.458 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.457+0000 7fa239f12540 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-01-10T06:59:35.555 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:35.554+0000 7fa239f12540 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-01-10T06:59:41.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-01-10T06:59:41.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T06:59:41.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T06:59:41.528 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T06:59:41.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T06:59:42.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T06:59:42.207 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T06:59:42.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T06:59:42.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T06:59:42.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T06:59:42.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T06:59:42.228 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:42.227+0000 7f2fff2fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:59:42.230 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:42.229+0000 7f2fff2fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:59:42.230 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:42.230+0000 7f2fff2fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:42.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T06:59:42.726 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T06:59:42.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:42.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:42.754 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:42.754+0000 7f2fff2fb540 -1 Falling back to public interface 2024-01-10T06:59:43.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:44.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:44.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:44.194 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T06:59:44.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T06:59:44.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:44.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:44.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:45.299 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T06:59:45.299+0000 7f2fff2fb540 -1 osd.0 0 log_to_monitors true 2024-01-10T06:59:45.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:45.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:45.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T06:59:45.715 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T06:59:45.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:45.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:46.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:47.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:47.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:47.209 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T06:59:47.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T06:59:47.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:47.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:47.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:48.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:48.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:48.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T06:59:48.667 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T06:59:48.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:48.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:49.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:50.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:50.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:50.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T06:59:50.242 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T06:59:50.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:50.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:50.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:51.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:51.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:51.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T06:59:51.705 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T06:59:51.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:51.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:52.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:53.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:53.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:53.186 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T06:59:53.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T06:59:53.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:53.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:53.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:54.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:54.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:54.641 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T06:59:54.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T06:59:54.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:54.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:55.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:56.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:56.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:56.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T06:59:56.174 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T06:59:56.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:56.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:56.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:57.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:57.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:57.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T06:59:57.661 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T06:59:57.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:57.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T06:59:58.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T06:59:59.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T06:59:59.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T06:59:59.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T06:59:59.194 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T06:59:59.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T06:59:59.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:00.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:01.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:01.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:01.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:00:01.055 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:00:01.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:01.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:01.738 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:02.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:02.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:02.740 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:00:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:00:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:03.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:04.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:04.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:04.648 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:00:04.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:00:04.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:04.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:05.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:06.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:06.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:06.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:00:06.400 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:00:06.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:06.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:07.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:08.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:08.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:08.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:00:08.443 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:00:08.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:08.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:09.800 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:10.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:10.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:10.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:00:10.802 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:00:10.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:10.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:11.334 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/65138,v1:127.0.0.1:6803/65138] [v2:127.0.0.1:6804/65138,v1:127.0.0.1:6805/65138] exists,up f83cefb6-c09f-4461-84b4-4e039534954b 2024-01-10T07:00:11.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:00:11.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:00:11.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:11.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:00:11.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-01-10T07:00:11.356 INFO:tasks.workunit.client.0.smithi057.stdout:starting mds.a at 2024-01-10T07:00:11.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-01-10T07:00:12.128 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'meta' already exists 2024-01-10T07:00:12.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-01-10T07:00:12.797 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'data' already exists 2024-01-10T07:00:12.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-01-10T07:00:13.895 INFO:tasks.workunit.client.0.smithi057.stderr:filesystem 'myfs' already exists 2024-01-10T07:00:13.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-01-10T07:00:18.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-01-10T07:00:19.331 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-01-10T07:00:19.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-01-10T07:00:19.728 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-01-10T07:00:19.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-01-10T07:00:20.437 INFO:tasks.workunit.client.0.smithi057.stderr:should be safe to stop [dne] 2024-01-10T07:00:20.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-01-10T07:00:20.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-01-10T07:00:20.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-01-10T07:00:20.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-01-10T07:00:20.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:00:20.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-01-10T07:00:20.468 INFO:tasks.workunit.client.0.smithi057.stdout:starting mds.b at 2024-01-10T07:00:20.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-01-10T07:00:25.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-01-10T07:00:26.113 INFO:tasks.workunit.client.0.smithi057.stderr:should be safe to stop [a] 2024-01-10T07:00:26.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-01-10T07:00:26.719 INFO:tasks.workunit.client.0.smithi057.stderr:should be safe to stop [b] 2024-01-10T07:00:26.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-01-10T07:00:27.116 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-01-10T07:00:27.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-01-10T07:00:27.739 INFO:tasks.workunit.client.0.smithi057.stderr:should be safe to stop [a,dne1,dne2] 2024-01-10T07:00:27.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-01-10T07:00:28.422 INFO:tasks.workunit.client.0.smithi057.stderr:should be safe to stop [b,dne] 2024-01-10T07:00:28.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-01-10T07:00:28.857 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-01-10T07:00:28.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-01-10T07:00:29.470 INFO:tasks.workunit.client.0.smithi057.stderr:should be safe to stop [dne1,dne2] 2024-01-10T07:00:29.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-01-10T07:00:29.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:00:29.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:00:29.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:00:29.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:00:29.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:00:29.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:00:29.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:00:29.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:00:29.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:00:29.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:00:29.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:00:29.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:00:29.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:00:29.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:00:29.699 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:00:29.700 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:00:29.700 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:00:29.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:00:29.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:00:29.701 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:00:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:00:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:00:29.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:00:29.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:00:29.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:00:29.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:00:29.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:00:29.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:00:29.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:29.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:29.711 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:00:29.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:00:29.714 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:00:29.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:00:29.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:00:29.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:00:29.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:00:29.714 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:00:29.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:00:29.715 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:00:29.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:00:29.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:00:29.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:00:29.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:00:29.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:00:29.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:00:29.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:00:29.719 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:00:29.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:00:29.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:00:29.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:00:29.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:29.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:29.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:00:29.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:00:29.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:00:29.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T07:00:29.723 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:00:29.723 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:29.723 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:29.723 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T07:00:29.725 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T07:00:29.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T07:00:30.375 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:00:30.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T07:00:30.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:00:30.681 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T07:00:30.783 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:00:30.783 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:00:30.783 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:00:30.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T07:00:30.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-01-10T07:00:30.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-01-10T07:00:30.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:00:30.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:00:30.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:00:30.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-01-10T07:00:30.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:00:31.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:00:31.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:00:31.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T07:00:31.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:00:31.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-01-10T07:00:31.841 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:00:31.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=273ba073-2ec1-40fd-8b82-b7d8e00d44e4 2024-01-10T07:00:31.842 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 273ba073-2ec1-40fd-8b82-b7d8e00d44e4 2024-01-10T07:00:31.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 273ba073-2ec1-40fd-8b82-b7d8e00d44e4' 2024-01-10T07:00:31.842 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:00:31.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPQJ5lRzHlMhAAndHORSz91ZxYbRv4hw5Law== 2024-01-10T07:00:31.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPQJ5lRzHlMhAAndHORSz91ZxYbRv4hw5Law=="}' 2024-01-10T07:00:31.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 273ba073-2ec1-40fd-8b82-b7d8e00d44e4 -i td/ok-to-stop/0/new.json 2024-01-10T07:00:32.331 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:00:32.342 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-01-10T07:00:32.343 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPQJ5lRzHlMhAAndHORSz91ZxYbRv4hw5Law== --osd-uuid 273ba073-2ec1-40fd-8b82-b7d8e00d44e4 2024-01-10T07:00:32.366 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.365+0000 7fadf372b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:00:32.368 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.367+0000 7fadf372b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:00:32.369 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.368+0000 7fadf372b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:00:32.370 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.369+0000 7fadf372b540 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-01-10T07:00:32.370 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.369+0000 7fadf372b540 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-01-10T07:00:32.370 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.369+0000 7fadf372b540 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-01-10T07:00:32.450 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:32.449+0000 7fadf372b540 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-01-10T07:00:38.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-01-10T07:00:38.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:00:38.452 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:00:38.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:00:38.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:00:39.115 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:00:39.116 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:00:39.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:00:39.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:00:39.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:00:39.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:00:39.137 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:39.136+0000 7fb473456540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:00:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:39.138+0000 7fb473456540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:00:39.140 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:39.139+0000 7fb473456540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:39.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:00:39.563 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:00:39.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:39.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:39.669 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:39.668+0000 7fb473456540 -1 Falling back to public interface 2024-01-10T07:00:40.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:41.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:41.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:41.073 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:00:41.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:00:41.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:41.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:41.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:42.244 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:00:42.243+0000 7fb473456540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:00:42.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:42.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:42.550 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:00:42.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:00:42.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:42.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:43.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:44.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:44.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:44.053 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:00:44.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:00:44.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:44.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:44.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:45.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:45.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:45.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:00:45.511 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:00:45.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:45.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:46.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:47.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:47.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:47.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:00:47.118 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:00:47.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:47.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:47.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:48.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:48.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:48.581 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:00:48.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:00:48.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:48.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:49.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:50.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:50.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:50.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:00:50.158 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:00:50.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:50.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:50.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:51.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:51.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:51.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:00:51.617 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:00:51.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:51.618 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:52.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:53.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:53.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:53.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:00:53.141 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:00:53.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:53.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:53.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:54.594 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:54.594 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:54.595 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:00:54.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:00:54.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:54.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:55.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:56.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:56.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:56.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:00:56.164 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:00:56.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:56.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:57.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:58.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:58.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:58.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:00:58.044 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:00:58.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:58.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:00:58.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:00:59.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:00:59.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:00:59.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:00:59.753 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:00:59.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:00:59.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:01:00.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:01.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:01.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:01.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:01:01.582 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:01:01.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:01.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:01:02.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:03.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:03.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:03.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:01:03.201 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:01:03.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:03.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:01:03.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:04.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:04.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:04.921 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:01:04.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:01:04.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:04.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:01:05.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:06.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:06.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:06.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:01:06.835 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:01:06.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:06.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:01:07.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:08.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:08.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:08.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:01:08.289 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:01:08.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:08.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:01:08.865 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/67645,v1:127.0.0.1:6803/67645] [v2:127.0.0.1:6804/67645,v1:127.0.0.1:6805/67645] exists,up 273ba073-2ec1-40fd-8b82-b7d8e00d44e4 2024-01-10T07:01:08.869 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-01-10T07:01:08.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-01-10T07:01:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a95ad4f-84e6-40b7-87e4-b5ac0c8e1d04 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1a95ad4f-84e6-40b7-87e4-b5ac0c8e1d04' 2024-01-10T07:01:08.872 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 1a95ad4f-84e6-40b7-87e4-b5ac0c8e1d04 2024-01-10T07:01:08.873 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:01:08.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0QJ5lI0yZNBAAm4Wr/AEKeheW9SJ+rxqOWw== 2024-01-10T07:01:08.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0QJ5lI0yZNBAAm4Wr/AEKeheW9SJ+rxqOWw=="}' 2024-01-10T07:01:08.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a95ad4f-84e6-40b7-87e4-b5ac0c8e1d04 -i td/ok-to-stop/1/new.json 2024-01-10T07:01:09.532 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:01:09.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-01-10T07:01:09.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0QJ5lI0yZNBAAm4Wr/AEKeheW9SJ+rxqOWw== --osd-uuid 1a95ad4f-84e6-40b7-87e4-b5ac0c8e1d04 2024-01-10T07:01:09.568 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.567+0000 7fdf05f29540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:09.570 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.569+0000 7fdf05f29540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:09.570 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.570+0000 7fdf05f29540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:09.571 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.571+0000 7fdf05f29540 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-01-10T07:01:09.571 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.571+0000 7fdf05f29540 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-01-10T07:01:09.571 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.571+0000 7fdf05f29540 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-01-10T07:01:09.656 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:09.656+0000 7fdf05f29540 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-01-10T07:01:16.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-01-10T07:01:16.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:01:16.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:01:16.464 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:01:16.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:01:17.084 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:01:17.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:01:17.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:01:17.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:01:17.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:01:17.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:01:17.105 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:17.105+0000 7fbebaf4d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:17.107 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:17.107+0000 7fbebaf4d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:17.108 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:17.108+0000 7fbebaf4d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:17.630 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:17.630+0000 7fbebaf4d540 -1 Falling back to public interface 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:01:17.636 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:01:17.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:17.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:18.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:19.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:19.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:19.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:01:19.077 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:01:19.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:19.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:19.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:20.302 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:20.302+0000 7fbebaf4d540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:01:20.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:20.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:20.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:01:20.584 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:01:20.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:20.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:21.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:22.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:22.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:22.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:01:22.054 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:01:22.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:22.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:22.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:23.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:23.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:23.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:01:23.512 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:01:23.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:23.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:23.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:24.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:24.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:24.972 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:01:24.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:01:24.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:24.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:25.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:26.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:26.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:26.515 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:01:26.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:01:26.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:26.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:26.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:27.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:27.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:27.988 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:01:27.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:01:27.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:27.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:28.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:29.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:29.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:29.580 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:01:29.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:01:29.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:29.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:30.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:31.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:31.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:31.262 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:01:31.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:01:31.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:31.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:31.723 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:32.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:32.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:32.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:01:32.724 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:01:32.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:32.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:33.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:34.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:34.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:34.341 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:01:34.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:01:34.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:34.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:34.967 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:35.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:35.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:35.969 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:01:35.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:01:35.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:35.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:36.570 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:37.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:37.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:37.572 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:01:37.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:01:37.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:37.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:38.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:39.551 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:01:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:01:39.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:39.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:40.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:41.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:41.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:41.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:01:41.402 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:01:41.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:41.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:42.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:43.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:43.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:43.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:01:43.209 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:01:43.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:43.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:44.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:45.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:45.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:45.151 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:01:45.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:01:45.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:45.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:45.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:46.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:46.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:46.782 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:01:46.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:01:46.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:46.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:01:47.380 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/69467,v1:127.0.0.1:6811/69467] [v2:127.0.0.1:6812/69467,v1:127.0.0.1:6813/69467] exists,up 1a95ad4f-84e6-40b7-87e4-b5ac0c8e1d04 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:01:47.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:01:47.385 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:01:47.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a27a7d9-f427-487e-b2b9-e54855731a92 2024-01-10T07:01:47.387 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 4a27a7d9-f427-487e-b2b9-e54855731a92 2024-01-10T07:01:47.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4a27a7d9-f427-487e-b2b9-e54855731a92' 2024-01-10T07:01:47.387 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:01:47.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbQJ5ljReyFxAAE/RR+ny+EJLiuv/ynp4+ow== 2024-01-10T07:01:47.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbQJ5ljReyFxAAE/RR+ny+EJLiuv/ynp4+ow=="}' 2024-01-10T07:01:47.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a27a7d9-f427-487e-b2b9-e54855731a92 -i td/ok-to-stop/2/new.json 2024-01-10T07:01:48.012 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:01:48.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-01-10T07:01:48.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbQJ5ljReyFxAAE/RR+ny+EJLiuv/ynp4+ow== --osd-uuid 4a27a7d9-f427-487e-b2b9-e54855731a92 2024-01-10T07:01:48.049 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.047+0000 7f547c745540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:48.050 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.049+0000 7f547c745540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:48.051 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.050+0000 7f547c745540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:48.052 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.050+0000 7f547c745540 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-01-10T07:01:48.052 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.050+0000 7f547c745540 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-01-10T07:01:48.052 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.050+0000 7f547c745540 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-01-10T07:01:48.173 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:48.173+0000 7f547c745540 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-01-10T07:01:54.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-01-10T07:01:54.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:01:54.945 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T07:01:54.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T07:01:54.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:01:55.566 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T07:01:55.566 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T07:01:55.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:01:55.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:01:55.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:01:55.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:01:55.587 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:55.587+0000 7f3f8b794540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:55.600 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:55.600+0000 7f3f8b794540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:55.601 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:55.601+0000 7f3f8b794540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:01:56.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T07:01:56.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:01:56.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T07:01:56.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:01:56.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:01:56.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:56.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:01:56.067 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:01:56.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:56.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:01:56.134 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:56.134+0000 7f3f8b794540 -1 Falling back to public interface 2024-01-10T07:01:56.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:57.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:57.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:57.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:01:57.576 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:01:57.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:57.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:01:58.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:01:58.757 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:01:58.756+0000 7f3f8b794540 -1 osd.2 0 log_to_monitors true 2024-01-10T07:01:59.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:01:59.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:01:59.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:01:59.073 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:01:59.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:01:59.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:01:59.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:00.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:00.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:00.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:02:00.764 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:02:00.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:00.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:01.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:02.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:02.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:02.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:02:02.465 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:02:02.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:02.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:03.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:04.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:04.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:04.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:02:04.205 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:02:04.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:04.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:04.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:05.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:05.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:05.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:02:05.936 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:02:05.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:05.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:06.389 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:07.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:07.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:07.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:02:07.391 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:02:07.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:07.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:07.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:08.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:08.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:08.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:02:08.956 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:02:08.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:08.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:09.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:10.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:10.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:10.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:02:10.503 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:02:10.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:10.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:10.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:11.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:11.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:11.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:02:11.966 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:02:11.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:11.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:12.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:13.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:13.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:13.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:02:13.450 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:02:13.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:13.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:14.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:15.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:15.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:15.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:02:15.077 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:02:15.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:15.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:15.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:16.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:16.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:16.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:02:16.750 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:02:16.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:16.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:17.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:18.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:18.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:18.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:02:18.445 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:02:18.445 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:18.445 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:19.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:20.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:20.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:20.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:02:20.051 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:02:20.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:20.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:21.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:22.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:22.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:22.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:02:22.056 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:02:22.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:22.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:22.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:23.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:23.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:23.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:02:23.761 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:02:23.761 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:23.761 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:24.738 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:25.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:25.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:25.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:02:25.739 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:02:25.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:25.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:02:26.237 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/71277,v1:127.0.0.1:6819/71277] [v2:127.0.0.1:6820/71277,v1:127.0.0.1:6821/71277] exists,up 4a27a7d9-f427-487e-b2b9-e54855731a92 2024-01-10T07:02:26.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:02:26.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:02:26.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:02:26.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:02:26.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-01-10T07:02:26.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c1c416b-8e93-4651-b227-fbf9bc9e11e6 2024-01-10T07:02:26.244 INFO:tasks.workunit.client.0.smithi057.stdout:add osd3 4c1c416b-8e93-4651-b227-fbf9bc9e11e6 2024-01-10T07:02:26.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 4c1c416b-8e93-4651-b227-fbf9bc9e11e6' 2024-01-10T07:02:26.245 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:02:26.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACQZ5lR4IpDxAA3eFbY+U1UBOog5jmOaC7vA== 2024-01-10T07:02:26.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACQZ5lR4IpDxAA3eFbY+U1UBOog5jmOaC7vA=="}' 2024-01-10T07:02:26.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c1c416b-8e93-4651-b227-fbf9bc9e11e6 -i td/ok-to-stop/3/new.json 2024-01-10T07:02:26.836 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:02:26.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-01-10T07:02:26.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACQZ5lR4IpDxAA3eFbY+U1UBOog5jmOaC7vA== --osd-uuid 4c1c416b-8e93-4651-b227-fbf9bc9e11e6 2024-01-10T07:02:26.873 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.873+0000 7f966d01f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:02:26.874 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.874+0000 7f966d01f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:02:26.875 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.875+0000 7f966d01f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:02:26.876 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.876+0000 7f966d01f540 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-01-10T07:02:26.876 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.876+0000 7f966d01f540 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-01-10T07:02:26.876 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.876+0000 7f966d01f540 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-01-10T07:02:26.976 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:26.976+0000 7f966d01f540 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-01-10T07:02:33.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-01-10T07:02:33.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:02:33.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-01-10T07:02:33.374 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd3 key to auth repository 2024-01-10T07:02:33.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:02:33.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-01-10T07:02:33.992 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.3 2024-01-10T07:02:33.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:02:33.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:02:33.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:02:33.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:02:34.014 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:34.013+0000 7f0d89576540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:02:34.016 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:34.016+0000 7f0d89576540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:02:34.017 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:34.016+0000 7f0d89576540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:02:34.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-01-10T07:02:34.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:02:34.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-01-10T07:02:34.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:02:34.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:02:34.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:34.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:02:34.475 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:02:34.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:34.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:34.539 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:34.539+0000 7f0d89576540 -1 Falling back to public interface 2024-01-10T07:02:34.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:35.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:35.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:35.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:02:35.976 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:02:35.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:35.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:36.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:37.382 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:02:37.382+0000 7f0d89576540 -1 osd.3 0 log_to_monitors true 2024-01-10T07:02:37.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:37.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:37.498 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:02:37.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:02:37.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:37.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:38.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:39.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:39.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:39.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:02:39.242 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:02:39.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:39.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:39.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:40.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:40.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:40.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:02:40.722 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:02:40.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:40.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:41.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:42.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:42.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:42.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:02:42.223 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:02:42.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:42.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:42.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:43.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:43.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:43.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:02:43.812 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:02:43.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:43.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:44.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:45.430 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:45.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:45.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:02:45.431 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:02:45.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:45.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:45.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:46.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:46.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:46.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:02:46.921 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:02:46.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:46.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:47.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:48.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:48.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:48.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:02:48.384 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:02:48.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:48.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:48.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:49.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:49.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:49.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:02:49.843 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:02:49.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:49.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:50.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:51.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:51.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:51.406 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:02:51.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:02:51.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:51.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:51.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:52.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:52.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:52.977 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:02:52.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:02:52.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:52.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:53.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:54.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:54.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:54.613 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:02:54.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:02:54.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:54.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:55.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:56.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:56.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:56.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:02:56.380 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:02:56.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:56.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:56.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:57.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:57.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:57.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:02:57.938 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:02:57.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:57.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:02:58.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:02:59.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:02:59.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:02:59.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:02:59.757 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:02:59.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:02:59.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:03:00.727 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:03:01.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:03:01.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:03:01.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:03:01.729 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:03:01.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:03:01.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:03:02.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:03:03.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:03:03.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:03:03.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:03:03.547 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:03:03.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:03:03.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:03:04.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:03:05.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:03:05.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:03:05.077 INFO:tasks.workunit.client.0.smithi057.stdout:19 2024-01-10T07:03:05.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-01-10T07:03:05.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:03:05.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-01-10T07:03:05.548 INFO:tasks.workunit.client.0.smithi057.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/73078,v1:127.0.0.1:6827/73078] [v2:127.0.0.1:6828/73078,v1:127.0.0.1:6829/73078] exists,up 4c1c416b-8e93-4651-b227-fbf9bc9e11e6 2024-01-10T07:03:05.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:03:05.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:03:05.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:03:05.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-01-10T07:03:06.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-01-10T07:03:07.584 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'ec' already exists 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:03:07.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:03:07.596 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:03:07.596 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:03:07.596 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:03:07.596 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:03:07.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:03:07.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:03:07.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:03:07.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:03:07.687 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:03:08.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:03:08.319 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T07:03:08.319 INFO:tasks.workunit.client.0.smithi057.stderr:2 2024-01-10T07:03:08.319 INFO:tasks.workunit.client.0.smithi057.stderr:3' 2024-01-10T07:03:08.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:03:08.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:08.320 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:03:08.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836510 2024-01-10T07:03:08.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836510 2024-01-10T07:03:08.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836510' 2024-01-10T07:03:08.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:08.582 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:03:08.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672986 2024-01-10T07:03:08.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672986 2024-01-10T07:03:08.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836510 1-42949672986' 2024-01-10T07:03:08.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:08.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T07:03:08.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509456 2024-01-10T07:03:08.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509456 2024-01-10T07:03:08.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836510 1-42949672986 2-64424509456' 2024-01-10T07:03:08.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:08.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-01-10T07:03:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=81604378630 2024-01-10T07:03:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 81604378630 2024-01-10T07:03:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836510 1-42949672986 2-64424509456 3-81604378630' 2024-01-10T07:03:09.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:09.038 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836510 2024-01-10T07:03:09.038 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:09.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:03:09.039 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836510 2024-01-10T07:03:09.039 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:09.040 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836510 2024-01-10T07:03:09.040 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-01-10T07:03:09.041 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836510 2024-01-10T07:03:09.043 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:03:09.608 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836508 -lt 21474836510 2024-01-10T07:03:09.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:03:10.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:03:10.610 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:03:11.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836511 -lt 21474836510 2024-01-10T07:03:11.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:11.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672986 2024-01-10T07:03:11.132 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:11.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:03:11.133 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672986 2024-01-10T07:03:11.133 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:11.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672986 2024-01-10T07:03:11.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-01-10T07:03:11.134 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672986 2024-01-10T07:03:11.134 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:03:11.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672986 -lt 42949672986 2024-01-10T07:03:11.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:11.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509456 2024-01-10T07:03:11.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:11.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T07:03:11.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509456 2024-01-10T07:03:11.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:11.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509456 2024-01-10T07:03:11.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-01-10T07:03:11.577 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509456 2024-01-10T07:03:11.577 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T07:03:12.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509456 -lt 64424509456 2024-01-10T07:03:12.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:12.035 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-81604378630 2024-01-10T07:03:12.035 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:12.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2024-01-10T07:03:12.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-81604378630 2024-01-10T07:03:12.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:12.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=81604378630 2024-01-10T07:03:12.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2024-01-10T07:03:12.038 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.3 seq 81604378630 2024-01-10T07:03:12.038 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2024-01-10T07:03:12.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 81604378631 -lt 81604378630 2024-01-10T07:03:12.481 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:03:12.481 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:12.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:13.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-01-10T07:03:13.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:03:13.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:03:13.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:03:13.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:03:13.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:03:13.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:03:13.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:13.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-01-10T07:03:13.573 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:03:13.573 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:13.573 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:14.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-01-10T07:03:14.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' -1 2024-01-10T07:03:14.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-01-10T07:03:14.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=11 2024-01-10T07:03:14.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:03:14.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-01-10T07:03:14.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:03:14.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:03:14.284 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:03:14.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:03:14.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:03:14.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:03:14.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:03:14.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:14.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-01-10T07:03:14.755 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:03:14.755 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:14.755 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:15.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-01-10T07:03:15.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:03:15.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:03:15.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-01-10T07:03:16.263 INFO:tasks.workunit.client.0.smithi057.stderr:set pool 1 min_size to 3 2024-01-10T07:03:16.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-01-10T07:03:16.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:03:16.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:03:16.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:03:16.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:03:16.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:03:16.276 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:03:16.276 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:03:16.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:03:16.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:03:16.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:03:16.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:03:16.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:03:16.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:03:16.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:03:16.371 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:03:16.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:03:16.833 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T07:03:16.833 INFO:tasks.workunit.client.0.smithi057.stderr:2 2024-01-10T07:03:16.833 INFO:tasks.workunit.client.0.smithi057.stderr:3' 2024-01-10T07:03:16.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:03:16.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:16.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:03:17.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836514 2024-01-10T07:03:17.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836514 2024-01-10T07:03:17.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514' 2024-01-10T07:03:17.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:17.004 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:03:17.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672989 2024-01-10T07:03:17.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672989 2024-01-10T07:03:17.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514 1-42949672989' 2024-01-10T07:03:17.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:17.142 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T07:03:17.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509460 2024-01-10T07:03:17.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509460 2024-01-10T07:03:17.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514 1-42949672989 2-64424509460' 2024-01-10T07:03:17.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:17.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-01-10T07:03:17.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=81604378634 2024-01-10T07:03:17.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 81604378634 2024-01-10T07:03:17.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836514 1-42949672989 2-64424509460 3-81604378634' 2024-01-10T07:03:17.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:17.412 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836514 2024-01-10T07:03:17.412 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:17.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:03:17.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836514 2024-01-10T07:03:17.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:17.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836514 2024-01-10T07:03:17.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-01-10T07:03:17.414 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836514 2024-01-10T07:03:17.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:03:17.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836512 -lt 21474836514 2024-01-10T07:03:17.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:03:18.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:03:18.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:03:19.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836514 -lt 21474836514 2024-01-10T07:03:19.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:19.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672989 2024-01-10T07:03:19.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:19.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:03:19.318 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672989 2024-01-10T07:03:19.318 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:19.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672989 2024-01-10T07:03:19.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-01-10T07:03:19.319 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672989 2024-01-10T07:03:19.319 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:03:19.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672990 -lt 42949672989 2024-01-10T07:03:19.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:19.763 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509460 2024-01-10T07:03:19.763 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:19.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T07:03:19.764 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509460 2024-01-10T07:03:19.764 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:19.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509460 2024-01-10T07:03:19.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-01-10T07:03:19.766 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509460 2024-01-10T07:03:19.766 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T07:03:20.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509460 -lt 64424509460 2024-01-10T07:03:20.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:20.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-81604378634 2024-01-10T07:03:20.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:20.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2024-01-10T07:03:20.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-81604378634 2024-01-10T07:03:20.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:20.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=81604378634 2024-01-10T07:03:20.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2024-01-10T07:03:20.278 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.3 seq 81604378634 2024-01-10T07:03:20.278 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2024-01-10T07:03:20.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 81604378635 -lt 81604378634 2024-01-10T07:03:20.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:03:20.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:20.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:03:21.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:21.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-01-10T07:03:21.773 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:03:21.773 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:21.773 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:22.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-01-10T07:03:22.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:03:22.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:03:22.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-01-10T07:03:22.841 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:22.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-01-10T07:03:23.281 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:23.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-01-10T07:03:23.726 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:23.738 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-01-10T07:03:24.173 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:24.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-01-10T07:03:24.486 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:24.487 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-01-10T07:03:24.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-01-10T07:03:24.788 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:24.790 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-01-10T07:03:24.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-01-10T07:03:24.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-01-10T07:03:25.242 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:25.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-01-10T07:03:25.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-01-10T07:03:25.688 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:25.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-01-10T07:03:26.822 INFO:tasks.workunit.client.0.smithi057.stderr:set pool 1 min_size to 2 2024-01-10T07:03:26.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-01-10T07:03:26.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:03:26.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:03:26.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:03:26.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:03:26.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:03:26.834 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:03:26.835 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:03:26.835 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:03:26.835 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:03:26.835 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:03:26.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:03:26.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:03:26.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:03:26.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:03:26.933 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:03:27.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:03:27.698 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T07:03:27.698 INFO:tasks.workunit.client.0.smithi057.stderr:2 2024-01-10T07:03:27.698 INFO:tasks.workunit.client.0.smithi057.stderr:3' 2024-01-10T07:03:27.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:03:27.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:27.699 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:03:27.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836518 2024-01-10T07:03:27.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836518 2024-01-10T07:03:27.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518' 2024-01-10T07:03:27.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:27.833 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:03:28.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672994 2024-01-10T07:03:28.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672994 2024-01-10T07:03:28.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518 1-42949672994' 2024-01-10T07:03:28.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:28.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T07:03:28.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509464 2024-01-10T07:03:28.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509464 2024-01-10T07:03:28.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518 1-42949672994 2-64424509464' 2024-01-10T07:03:28.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:28.177 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-01-10T07:03:28.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=81604378638 2024-01-10T07:03:28.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 81604378638 2024-01-10T07:03:28.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518 1-42949672994 2-64424509464 3-81604378638' 2024-01-10T07:03:28.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:28.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836518 2024-01-10T07:03:28.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:28.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:03:28.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836518 2024-01-10T07:03:28.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:28.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836518 2024-01-10T07:03:28.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-01-10T07:03:28.339 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836518 2024-01-10T07:03:28.339 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:03:28.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836518 -lt 21474836518 2024-01-10T07:03:28.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:28.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672994 2024-01-10T07:03:28.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:28.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:03:28.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672994 2024-01-10T07:03:28.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:28.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672994 2024-01-10T07:03:28.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-01-10T07:03:28.814 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672994 2024-01-10T07:03:28.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:03:29.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672993 -lt 42949672994 2024-01-10T07:03:29.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:03:30.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:03:30.271 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:03:30.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672994 -lt 42949672994 2024-01-10T07:03:30.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:30.732 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509464 2024-01-10T07:03:30.732 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:30.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T07:03:30.733 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509464 2024-01-10T07:03:30.733 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:30.734 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509464 2024-01-10T07:03:30.734 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-01-10T07:03:30.734 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509464 2024-01-10T07:03:30.735 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T07:03:31.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509464 -lt 64424509464 2024-01-10T07:03:31.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:31.180 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-81604378638 2024-01-10T07:03:31.180 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:31.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2024-01-10T07:03:31.181 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-81604378638 2024-01-10T07:03:31.181 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:31.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=81604378638 2024-01-10T07:03:31.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2024-01-10T07:03:31.182 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.3 seq 81604378638 2024-01-10T07:03:31.182 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2024-01-10T07:03:31.676 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 81604378639 -lt 81604378638 2024-01-10T07:03:31.676 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:03:31.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:31.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:32.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-01-10T07:03:32.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:03:32.287 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:03:32.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:03:32.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:03:32.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:03:32.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:03:32.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:32.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2024-01-10T07:03:32.805 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:03:32.805 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:32.806 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:33.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2024-01-10T07:03:33.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' -1 2024-01-10T07:03:33.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-01-10T07:03:33.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=19 2024-01-10T07:03:33.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:03:33.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:03:33.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:33.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2024-01-10T07:03:33.960 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:03:33.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:33.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:34.564 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2024-01-10T07:03:34.564 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' 19 2024-01-10T07:03:34.564 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-01-10T07:03:34.564 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-01-10T07:03:34.565 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-01-10T07:03:34.565 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-01-10T07:03:34.565 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-01-10T07:03:34.565 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-01-10T07:03:34.565 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-01-10T07:03:35.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-01-10T07:03:35.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-01-10T07:03:35.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-01-10T07:03:35.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:03:35.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-01-10T07:03:35.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:03:35.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:03:35.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:03:35.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:03:35.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:03:35.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:03:35.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:03:35.413 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:35.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-01-10T07:03:35.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:03:35.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:35.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:36.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-01-10T07:03:36.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:03:36.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:03:36.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-01-10T07:03:37.001 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:37.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-01-10T07:03:37.448 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:37.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-01-10T07:03:37.756 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:37.757 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-01-10T07:03:37.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-01-10T07:03:38.060 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:38.061 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-01-10T07:03:38.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-01-10T07:03:38.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-01-10T07:03:38.502 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:38.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-01-10T07:03:38.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-01-10T07:03:38.942 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:38.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-01-10T07:03:38.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-01-10T07:03:39.431 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:39.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-01-10T07:03:39.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-01-10T07:03:39.873 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:39.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-01-10T07:03:39.875 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:39.875 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:39.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:39.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:39.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:40.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:40.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-01-10T07:03:41.013 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already down. 2024-01-10T07:03:41.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:03:41.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:03:41.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:03:41.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:03:41.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-01-10T07:03:41.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-01-10T07:03:41.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-01-10T07:03:41.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:03:41.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:03:41.735 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:03:41.735 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T07:03:41.736 INFO:tasks.workunit.client.0.smithi057.stderr:2 2024-01-10T07:03:41.736 INFO:tasks.workunit.client.0.smithi057.stderr:3' 2024-01-10T07:03:41.736 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:03:41.736 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:41.736 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:03:41.849 INFO:tasks.workunit.client.0.smithi057.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-01-10T07:03:41.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq= 2024-01-10T07:03:41.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z '' 2024-01-10T07:03:41.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: continue 2024-01-10T07:03:41.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:41.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:03:41.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672998 2024-01-10T07:03:41.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672998 2024-01-10T07:03:41.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 1-42949672998' 2024-01-10T07:03:41.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:41.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-01-10T07:03:42.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509469 2024-01-10T07:03:42.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509469 2024-01-10T07:03:42.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 1-42949672998 2-64424509469' 2024-01-10T07:03:42.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:03:42.147 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-01-10T07:03:42.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=81604378643 2024-01-10T07:03:42.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 81604378643 2024-01-10T07:03:42.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 1-42949672998 2-64424509469 3-81604378643' 2024-01-10T07:03:42.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:42.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672998 2024-01-10T07:03:42.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:42.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:03:42.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672998 2024-01-10T07:03:42.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:42.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672998 2024-01-10T07:03:42.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-01-10T07:03:42.283 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672998 2024-01-10T07:03:42.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:03:42.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672998 -lt 42949672998 2024-01-10T07:03:42.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:42.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509469 2024-01-10T07:03:42.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:42.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2024-01-10T07:03:42.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509469 2024-01-10T07:03:42.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:42.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509469 2024-01-10T07:03:42.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-01-10T07:03:42.790 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.2 seq 64424509469 2024-01-10T07:03:42.790 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T07:03:43.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509467 -lt 64424509469 2024-01-10T07:03:43.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:03:44.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:03:44.232 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2024-01-10T07:03:44.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509469 -lt 64424509469 2024-01-10T07:03:44.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:03:44.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-81604378643 2024-01-10T07:03:44.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:03:44.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2024-01-10T07:03:44.689 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-81604378643 2024-01-10T07:03:44.689 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:03:44.690 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=81604378643 2024-01-10T07:03:44.690 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 81604378643' 2024-01-10T07:03:44.690 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.3 seq 81604378643 2024-01-10T07:03:44.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2024-01-10T07:03:45.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 81604378643 -lt 81604378643 2024-01-10T07:03:45.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-01-10T07:03:45.159 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:45.159 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-01-10T07:03:45.760 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-01-10T07:03:46.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-01-10T07:03:46.226 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-01-10T07:03:46.226 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:03:46.227 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:03:46.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-01-10T07:03:46.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-01-10T07:03:46.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-01-10T07:03:46.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-01-10T07:03:47.342 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-01-10T07:03:47.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-01-10T07:03:47.788 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:47.800 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-01-10T07:03:48.101 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:48.103 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-01-10T07:03:48.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-01-10T07:03:48.404 INFO:tasks.workunit.client.0.smithi057.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-01-10T07:03:48.405 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-01-10T07:03:48.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:48.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:48.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:48.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:48.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:48.409 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:48.517 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:48.517 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:48.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:48.524 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:03:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:03:48.525 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:03:48.526 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:03:48.526 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:03:48.526 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:03:48.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:48.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:03:48.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:03:48.529 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:48.529 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:03:48.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:03:48.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:03:48.551 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:03:48.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:48.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:48.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:03:48.553 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:48.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:48.555 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:48.555 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:03:48.555 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:03:48.556 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:03:48.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:03:48.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:03:48.557 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:03:48.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:48.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:03:48.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:03:48.559 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:48.560 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:03:48.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:03:48.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:03:48.562 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:03:48.562 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:48.562 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:48.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:03:48.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:03:48.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:03:48.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T07:03:48.564 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:03:48.564 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:48.564 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:48.565 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:03:48.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T07:03:48.568 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T07:03:49.001 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:03:49.001 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:03:49.002 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:03:49.002 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:03:49.002 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:49.002 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:49.002 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:03:49.002 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T07:03:49.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:03:49.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:03:49.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:49.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:03:49.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:49.370 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:49.371 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:03:49.371 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:49.371 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T07:03:49.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:03:49.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:49.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:49.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T07:03:49.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-01-10T07:03:50.461 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a) after stopping [dne] 2024-01-10T07:03:50.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-01-10T07:03:50.679 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-01-10T07:03:50.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-01-10T07:03:50.875 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-01-10T07:03:50.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-01-10T07:03:51.072 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: removing mon.a would break quorum 2024-01-10T07:03:51.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-01-10T07:03:51.339 INFO:tasks.workunit.client.0.smithi057.stderr:mon.dne does not exist 2024-01-10T07:03:51.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:51.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:51.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:51.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:51.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:51.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:51.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:51.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:51.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:51.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:51.459 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:03:51.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:03:51.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:03:51.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:03:51.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:03:51.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:03:51.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:51.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:03:51.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:03:51.463 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:51.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:03:51.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:03:51.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:03:51.466 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:03:51.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:51.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:51.467 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:51.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:51.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:51.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:03:51.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:03:51.471 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:03:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:03:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:03:51.472 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:03:51.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:51.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:03:51.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:03:51.475 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:51.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:03:51.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:03:51.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:03:51.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:03:51.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:51.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:51.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:03:51.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:03:51.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:03:51.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T07:03:51.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:03:51.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:51.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:51.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-01-10T07:03:51.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T07:03:51.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:03:51.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:03:51.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:03:51.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:03:51.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T07:03:51.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T07:03:51.887 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:03:51.887 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:03:51.887 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:03:51.887 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:03:51.888 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:51.888 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:51.888 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:03:51.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T07:03:52.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:03:52.241 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:52.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:03:52.353 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:52.354 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:52.354 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:52.354 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:03:52.354 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:52.354 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T07:03:52.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-01-10T07:03:52.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:03:52.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:03:52.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-01-10T07:03:52.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:03:52.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-01-10T07:03:52.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-01-10T07:03:52.875 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:03:52.875 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:03:52.876 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:03:52.876 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:03:52.876 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:52.876 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:52.876 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:03:52.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-01-10T07:03:53.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:03:53.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-01-10T07:03:53.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:53.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:03:53.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:03:53.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:03:53.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:03:53.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:03:53.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:03:53.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:53.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:53.131 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:53.131 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:03:53.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:53.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get fsid 2024-01-10T07:03:53.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-01-10T07:03:53.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:53.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get mon_host 2024-01-10T07:03:53.343 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-01-10T07:03:56.606 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b) after stopping [dne] 2024-01-10T07:03:56.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:03:56.868 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-01-10T07:03:56.869 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:03:57.096 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-01-10T07:03:57.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-01-10T07:03:57.451 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-01-10T07:03:57.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:03:57.710 INFO:tasks.workunit.client.0.smithi057.stderr:adding another mon that is not yet online will not break quorum 2024-01-10T07:03:57.719 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:03:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:03:57.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:03:58.266 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:03:58.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-01-10T07:03:58.653 INFO:tasks.workunit.client.0.smithi057.stderr:mon.dne does not exist 2024-01-10T07:03:58.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:58.663 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:58.663 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:58.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:58.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:58.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:58.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:58.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:58.773 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:03:58.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:03:58.774 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:03:58.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:03:58.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:03:58.776 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:03:58.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:58.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:03:58.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:03:58.778 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:58.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:03:58.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:03:58.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:03:58.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:03:58.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:58.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:58.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:03:58.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:03:58.785 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:03:58.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:03:58.786 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:03:58.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:03:58.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:03:58.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:03:58.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:58.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:03:58.790 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:03:58.790 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:03:58.790 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:03:58.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:03:58.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:03:58.793 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:03:58.793 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:58.793 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:58.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:03:58.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:03:58.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:03:58.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T07:03:58.795 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:03:58.795 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:58.795 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:58.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T07:03:58.797 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:03:58.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:03:58.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:03:58.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:03:58.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T07:03:58.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T07:03:59.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:03:59.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:03:59.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:03:59.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:03:59.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:59.201 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:59.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:03:59.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T07:03:59.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:03:59.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:03:59.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:59.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:03:59.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:03:59.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:03:59.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:59.505 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:03:59.606 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:03:59.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-01-10T07:03:59.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-01-10T07:04:00.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:04:00.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:04:00.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:04:00.171 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:04:00.172 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:00.172 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:00.172 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:04:00.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-01-10T07:04:00.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:04:00.458 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-01-10T07:04:00.458 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:00.458 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:04:00.458 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:04:00.458 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:04:00.458 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:04:00.459 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:04:00.459 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:04:00.459 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:00.459 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:00.460 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:00.460 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:04:00.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:00.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get fsid 2024-01-10T07:04:00.566 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-01-10T07:04:00.566 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:00.566 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:04:00.566 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:04:00.566 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:00.567 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get mon_host 2024-01-10T07:04:00.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-01-10T07:04:00.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:04:00.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:04:00.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-01-10T07:04:00.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:04:00.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-01-10T07:04:00.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-01-10T07:04:01.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:04:01.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:04:01.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:04:01.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:04:01.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:01.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:01.119 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:04:01.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-01-10T07:04:01.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:04:01.396 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-01-10T07:04:01.397 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:01.397 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-01-10T07:04:01.397 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.c.asok 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:01.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.c.asok config get fsid 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-01-10T07:04:01.502 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.c.asok 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:01.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.c.asok config get mon_host 2024-01-10T07:04:01.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-01-10T07:04:01.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:04:01.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-01-10T07:04:01.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:04:01.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-01-10T07:04:01.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:04:01.604 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:04:01.605 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870301 2024-01-10T07:04:01.605 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:01.605 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870241 -lt 1704870301 ]] 2024-01-10T07:04:01.605 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:01.605 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:04.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:04.848 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:04.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:04.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:04.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:04:04.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:04:04.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:04:04.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:04:04.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:04:04.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-01-10T07:04:05.361 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,c) after stopping [dne] 2024-01-10T07:04:05.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:04:05.827 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c) after stopping [a] 2024-01-10T07:04:05.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:04:06.164 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,c) after stopping [b] 2024-01-10T07:04:06.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:04:06.437 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b) after stopping [c] 2024-01-10T07:04:06.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-01-10T07:04:09.641 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-01-10T07:04:09.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-01-10T07:04:09.833 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-01-10T07:04:09.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-01-10T07:04:10.096 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-01-10T07:04:10.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:04:10.523 INFO:tasks.workunit.client.0.smithi057.stderr:adding another mon that is not yet online will not break quorum 2024-01-10T07:04:10.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:04:10.840 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:04:10.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:04:11.294 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:04:11.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:04:11.632 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.c 2024-01-10T07:04:11.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-01-10T07:04:11.642 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:04:11.642 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:04:11.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:04:11.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:04:11.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:04:11.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:04:11.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870311 2024-01-10T07:04:11.750 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:11.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870251 -lt 1704870311 ]] 2024-01-10T07:04:11.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:11.750 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:11.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:11.986 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:11.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:11.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:11.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:11.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:11.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:11.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870251 -lt 1704870311 ]] 2024-01-10T07:04:11.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:11.990 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:12.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:12.224 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:12.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:12.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:12.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:12.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:12.228 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:12.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870252 -lt 1704870311 ]] 2024-01-10T07:04:12.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:12.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:15.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:15.469 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:15.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:15.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:15.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:15.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:15.473 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:15.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870255 -lt 1704870311 ]] 2024-01-10T07:04:15.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:15.474 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:15.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:15.709 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:15.709 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:15.709 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:15.712 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:15.712 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:15.712 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:15.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870255 -lt 1704870311 ]] 2024-01-10T07:04:15.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:15.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:15.949 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:15.950 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:15.950 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:15.950 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:15.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:15.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:15.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:15.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870255 -lt 1704870311 ]] 2024-01-10T07:04:15.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:15.954 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:19.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:19.195 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.196 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:19.196 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:19.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:19.198 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:19.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870259 -lt 1704870311 ]] 2024-01-10T07:04:19.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:19.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:19.435 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:19.435 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.435 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:19.436 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:19.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:19.438 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:19.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870259 -lt 1704870311 ]] 2024-01-10T07:04:19.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:19.439 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:19.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:19.676 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.676 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:19.676 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:19.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:19.679 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:19.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870259 -lt 1704870311 ]] 2024-01-10T07:04:19.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:19.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:19.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:19.914 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.914 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:19.914 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:19.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:19.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:19.917 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:19.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870259 -lt 1704870311 ]] 2024-01-10T07:04:19.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:19.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:20.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:20.153 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:20.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:20.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:20.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:20.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:20.156 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:20.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870260 -lt 1704870311 ]] 2024-01-10T07:04:20.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-01-10T07:04:20.157 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:27.137 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:27.137 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:27.138 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-01-10T07:04:27.138 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:03:58.815030Z","created":"2024-01-10T07:03:58.815030Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:27.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:04:27.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:04:27.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:04:27.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:04:27.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:04:27.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:04:27.380 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-01-10T07:04:27.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:04:27.701 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,c) after stopping [b] 2024-01-10T07:04:27.710 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:04:27.909 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-01-10T07:04:27.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:04:28.186 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-01-10T07:04:28.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:04:28.463 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: removing mon.a would break quorum 2024-01-10T07:04:28.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:04:28.748 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:04:28.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:04:31.935 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: removing mon.c would break quorum 2024-01-10T07:04:31.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:04:31.938 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:04:31.938 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:04:31.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:04:31.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:04:31.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:04:32.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:04:32.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:04:32.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:04:32.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:04:32.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:04:32.045 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:04:32.046 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:04:32.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:04:32.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:04:32.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:04:32.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:04:32.050 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:04:32.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:04:32.051 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:04:32.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:04:32.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:04:32.053 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:04:32.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:04:32.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:04:32.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:04:32.055 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:04:32.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:04:32.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:04:32.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:04:32.061 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:04:32.061 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:32.061 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:32.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:04:32.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:04:32.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:04:32.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:04:32.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:04:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:04:32.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:04:32.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:04:32.067 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:04:32.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:04:32.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:04:32.068 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:04:32.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:04:32.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:04:32.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:04:32.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:04:32.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:04:32.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:04:32.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:04:32.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:04:32.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:32.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:32.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:04:32.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:04:32.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:04:32.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T07:04:32.075 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:04:32.076 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:32.076 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:32.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-01-10T07:04:32.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T07:04:32.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:04:32.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:04:32.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:04:32.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:04:32.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T07:04:32.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T07:04:32.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:04:32.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:04:32.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:04:32.498 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:04:32.498 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:32.499 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:32.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:04:32.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T07:04:32.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:04:32.821 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:04:32.821 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:32.822 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:04:32.822 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:04:32.822 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:04:32.822 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:04:32.822 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:04:32.823 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:04:32.823 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:32.823 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:32.823 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:32.823 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:04:32.823 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:32.824 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T07:04:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:04:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:04:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:04:32.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:04:32.934 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:32.935 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-01-10T07:04:33.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-01-10T07:04:33.437 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:04:33.437 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:04:33.437 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:04:33.438 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:04:33.438 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:33.438 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:33.438 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:04:33.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-01-10T07:04:33.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:04:33.740 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:04:33.741 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:33.741 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:33.741 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:33.742 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:04:33.742 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:33.742 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get fsid 2024-01-10T07:04:33.853 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:04:33.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:33.855 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get mon_host 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-01-10T07:04:33.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-01-10T07:04:34.302 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:04:34.302 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:04:34.302 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:04:34.302 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:04:34.302 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:34.302 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:34.303 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:04:34.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-01-10T07:04:34.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:04:34.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-01-10T07:04:34.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:34.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-01-10T07:04:34.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:04:34.586 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:04:34.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-01-10T07:04:34.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-01-10T07:04:34.587 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-01-10T07:04:34.587 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:34.587 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:34.588 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:34.588 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.c.asok 2024-01-10T07:04:34.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:34.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.c.asok config get fsid 2024-01-10T07:04:34.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-01-10T07:04:34.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:34.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.c.asok 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:34.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.c.asok config get mon_host 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-01-10T07:04:34.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-01-10T07:04:35.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:04:35.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:04:35.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:04:35.238 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:04:35.238 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:35.238 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:35.238 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:04:35.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-01-10T07:04:35.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:04:35.533 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-01-10T07:04:35.533 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:35.533 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-01-10T07:04:35.533 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:04:35.534 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:04:35.534 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-01-10T07:04:35.534 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.d.asok 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:35.535 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.d.asok config get fsid 2024-01-10T07:04:35.646 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-01-10T07:04:35.646 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:04:35.646 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-01-10T07:04:35.646 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:04:35.646 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.d.asok 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:04:35.647 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.d.asok config get mon_host 2024-01-10T07:04:35.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-01-10T07:04:35.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:04:35.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-01-10T07:04:35.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:04:35.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-01-10T07:04:35.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:04:35.749 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:04:35.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870335 2024-01-10T07:04:35.750 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:35.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870275 -lt 1704870335 ]] 2024-01-10T07:04:35.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:04:35.750 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:48.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:48.314 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:48.315 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:04:48.315 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:48.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:04:48.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:04:48.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:04:48.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:04:48.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:04:48.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-01-10T07:04:48.811 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,c,d) after stopping [dne] 2024-01-10T07:04:48.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:04:49.165 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-01-10T07:04:49.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:04:49.496 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-01-10T07:04:49.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:04:50.067 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-01-10T07:04:50.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-01-10T07:04:50.427 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-01-10T07:04:50.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-01-10T07:04:50.804 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-01-10T07:04:50.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-01-10T07:04:51.155 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-01-10T07:04:51.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:04:51.509 INFO:tasks.workunit.client.0.smithi057.stderr:adding another mon that is not yet online will not break quorum 2024-01-10T07:04:51.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:04:51.861 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:04:51.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:04:52.151 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:04:52.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:04:55.423 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.c 2024-01-10T07:04:55.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-01-10T07:04:55.434 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:04:55.434 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:04:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:04:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:04:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:04:55.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:04:55.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-01-10T07:04:55.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:04:55.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-01-10T07:04:55.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:04:55.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-01-10T07:04:55.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:04:55.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:04:55.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870355 2024-01-10T07:04:55.541 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:55.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870295 -lt 1704870355 ]] 2024-01-10T07:04:55.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:55.541 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:55.780 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:55.780 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:55.781 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:55.781 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:55.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:55.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:55.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:55.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870295 -lt 1704870355 ]] 2024-01-10T07:04:55.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:55.785 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:59.027 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.027 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:59.027 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:59.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:59.030 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:59.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870299 -lt 1704870355 ]] 2024-01-10T07:04:59.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:59.031 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:59.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:59.266 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.266 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:59.266 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:59.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:59.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:59.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870299 -lt 1704870355 ]] 2024-01-10T07:04:59.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:59.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:59.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:59.510 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.510 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:59.510 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:59.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:59.513 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:59.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870299 -lt 1704870355 ]] 2024-01-10T07:04:59.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:59.514 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:59.746 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:59.746 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:59.747 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:59.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:59.750 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:59.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870299 -lt 1704870355 ]] 2024-01-10T07:04:59.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:59.751 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:04:59.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:04:59.986 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:04:59.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:04:59.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:04:59.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:04:59.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:04:59.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870299 -lt 1704870355 ]] 2024-01-10T07:04:59.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:04:59.990 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:09.250 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:09.251 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:09.251 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:05:09.251 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:04:32.096407Z","created":"2024-01-10T07:04:32.096407Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:09.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:05:09.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:05:09.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:05:09.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:05:09.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:05:09.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:05:09.498 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-01-10T07:05:09.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:05:09.902 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-01-10T07:05:09.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:05:10.353 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-01-10T07:05:10.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-01-10T07:05:13.534 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-01-10T07:05:13.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:05:13.814 INFO:tasks.workunit.client.0.smithi057.stderr:adding another mon that is not yet online will not break quorum 2024-01-10T07:05:13.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:05:14.255 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:05:14.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:05:14.547 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:05:14.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:05:14.857 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.c 2024-01-10T07:05:14.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-01-10T07:05:15.134 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.d 2024-01-10T07:05:15.144 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:05:15.144 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:05:15.145 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:05:15.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:05:15.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:05:15.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:05:15.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:05:15.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:05:15.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:05:15.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:05:15.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:05:15.253 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:05:15.253 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:05:15.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:05:15.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:05:15.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:05:15.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:05:15.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:05:15.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:05:15.259 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:05:15.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:05:15.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:05:15.260 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:05:15.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:05:15.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:05:15.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:05:15.262 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:05:15.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:05:15.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:05:15.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:05:15.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:05:15.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:15.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:15.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:05:15.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:05:15.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:05:15.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:05:15.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:05:15.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:05:15.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:05:15.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:05:15.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:05:15.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:05:15.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:05:15.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:05:15.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:05:15.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:05:15.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:05:15.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:05:15.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:05:15.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:05:15.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:05:15.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:05:15.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:15.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:15.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:05:15.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:05:15.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:05:15.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-01-10T07:05:15.282 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:05:15.282 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:15.282 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:15.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64006 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=c4352b7c-e075-4d39-9959-a72f24818694 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-01-10T07:05:15.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:05:15.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-01-10T07:05:15.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:15.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:05:15.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:15.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get fsid 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:05:16.089 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:16.090 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:16.090 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:16.090 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.a.asok 2024-01-10T07:05:16.090 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:16.090 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.a.asok config get mon_host 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-01-10T07:05:16.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:16.661 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:05:16.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-01-10T07:05:17.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:05:17.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-01-10T07:05:17.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:17.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:05:17.006 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:05:17.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:05:17.007 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:05:17.007 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:05:17.007 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:05:17.008 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:17.008 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:17.008 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:17.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:05:17.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:17.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get fsid 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-01-10T07:05:17.115 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.b.asok 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:17.116 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.b.asok config get mon_host 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-01-10T07:05:17.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-01-10T07:05:17.615 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:05:17.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-01-10T07:05:17.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:05:17.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-01-10T07:05:17.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:17.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-01-10T07:05:17.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:05:17.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:05:17.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.c.asok 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:17.963 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.c.asok config get fsid 2024-01-10T07:05:18.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-01-10T07:05:18.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.c.asok 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:18.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.c.asok config get mon_host 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-01-10T07:05:18.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:05:18.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-01-10T07:05:18.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:05:18.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-01-10T07:05:18.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:18.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-01-10T07:05:18.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:05:18.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:05:18.900 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-01-10T07:05:18.900 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-01-10T07:05:18.900 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-01-10T07:05:18.901 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:18.901 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:18.901 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:18.901 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.d.asok 2024-01-10T07:05:18.901 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:18.901 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.d.asok config get fsid 2024-01-10T07:05:19.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.d.asok 2024-01-10T07:05:19.013 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:19.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.d.asok config get mon_host 2024-01-10T07:05:19.115 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-01-10T07:05:19.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-01-10T07:05:19.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:05:19.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-01-10T07:05:19.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:05:19.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-01-10T07:05:19.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-01-10T07:05:19.558 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:05:19.558 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:05:19.558 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:05:19.558 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:05:19.558 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:19.559 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:19.559 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64006/$cluster-$name.asok' 2024-01-10T07:05:19.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.64006/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-01-10T07:05:19.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:05:19.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-01-10T07:05:19.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:19.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-01-10T07:05:19.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:19.863 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.e.asok 2024-01-10T07:05:19.864 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:19.864 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.e.asok config get fsid 2024-01-10T07:05:19.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-01-10T07:05:19.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:05:19.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-01-10T07:05:19.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:05:19.975 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:05:19.975 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64006/ceph-mon.e.asok 2024-01-10T07:05:19.976 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:05:19.977 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64006/ceph-mon.e.asok config get mon_host 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:05:20.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:05:20.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870380 2024-01-10T07:05:20.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:20.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870320 -lt 1704870380 ]] 2024-01-10T07:05:20.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-01-10T07:05:20.081 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:32.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:32.350 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:32.350 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-01-10T07:05:32.350 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:32.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:05:32.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:05:32.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:05:32.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:05:32.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:05:32.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-01-10T07:05:32.798 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,c,d,e) after stopping [dne] 2024-01-10T07:05:32.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:05:36.061 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-01-10T07:05:36.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:05:36.675 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-01-10T07:05:36.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:05:36.998 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-01-10T07:05:37.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-01-10T07:05:40.241 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-01-10T07:05:40.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-01-10T07:05:40.586 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-01-10T07:05:40.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-01-10T07:05:41.314 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-01-10T07:05:41.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-01-10T07:05:41.759 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-01-10T07:05:41.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-01-10T07:05:41.999 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-01-10T07:05:42.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:05:42.527 INFO:tasks.workunit.client.0.smithi057.stderr:adding another mon that is not yet online will not break quorum 2024-01-10T07:05:42.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:05:42.883 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:05:42.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:05:46.167 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:05:46.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:05:46.677 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.c 2024-01-10T07:05:46.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-01-10T07:05:49.923 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.d 2024-01-10T07:05:49.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-01-10T07:05:53.167 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.e 2024-01-10T07:05:53.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-01-10T07:05:53.178 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:05:53.178 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:05:53.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:05:53.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:05:53.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-01-10T07:05:53.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:05:53.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:05:53.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870413 2024-01-10T07:05:53.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:53.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870353 -lt 1704870413 ]] 2024-01-10T07:05:53.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:53.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:53.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:53.519 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:53.520 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:53.520 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:53.522 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:53.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:53.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:53.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870353 -lt 1704870413 ]] 2024-01-10T07:05:53.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:53.524 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:56.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:56.765 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:56.766 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:56.766 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:56.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:56.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:56.769 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:56.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870356 -lt 1704870413 ]] 2024-01-10T07:05:56.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:56.770 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:57.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:57.004 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.004 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:57.005 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:57.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:57.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870357 -lt 1704870413 ]] 2024-01-10T07:05:57.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:57.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:57.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:57.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:57.246 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:57.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870357 -lt 1704870413 ]] 2024-01-10T07:05:57.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:57.247 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:57.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:57.481 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:57.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:57.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:57.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:57.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870357 -lt 1704870413 ]] 2024-01-10T07:05:57.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:57.486 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:57.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:57.721 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.722 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:57.722 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:57.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:57.725 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:57.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870357 -lt 1704870413 ]] 2024-01-10T07:05:57.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:57.726 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:57.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:57.961 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:57.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:57.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:57.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:57.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:57.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870357 -lt 1704870413 ]] 2024-01-10T07:05:57.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:57.965 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:58.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:58.201 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:58.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:58.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:58.205 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:58.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870358 -lt 1704870413 ]] 2024-01-10T07:05:58.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:58.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:58.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:58.440 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.440 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:58.440 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:58.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:58.443 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:58.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870358 -lt 1704870413 ]] 2024-01-10T07:05:58.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:58.444 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:58.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:58.680 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:58.680 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:58.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:58.683 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:58.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870358 -lt 1704870413 ]] 2024-01-10T07:05:58.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:58.684 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:58.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:58.918 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:58.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:58.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:58.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:58.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:58.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870358 -lt 1704870413 ]] 2024-01-10T07:05:58.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:58.922 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:59.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:59.157 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:59.157 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:59.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:59.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:59.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:59.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:59.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870359 -lt 1704870413 ]] 2024-01-10T07:05:59.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:59.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:59.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:59.395 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:59.395 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:59.395 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:59.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:59.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:59.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:59.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870359 -lt 1704870413 ]] 2024-01-10T07:05:59.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:59.399 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:05:59.636 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:05:59.636 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:59.636 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:05:59.636 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:05:59.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:05:59.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:05:59.639 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:05:59.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870359 -lt 1704870413 ]] 2024-01-10T07:05:59.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-01-10T07:05:59.640 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:07.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:07.335 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:07.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-01-10T07:06:07.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:07.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:06:07.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:06:07.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:06:07.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:06:07.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:06:07.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:06:07.571 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-01-10T07:06:07.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:06:08.059 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-01-10T07:06:08.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:06:08.542 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-01-10T07:06:08.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-01-10T07:06:11.787 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-01-10T07:06:11.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-01-10T07:06:12.405 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-01-10T07:06:12.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:06:12.820 INFO:tasks.workunit.client.0.smithi057.stderr:adding another mon that is not yet online will not break quorum 2024-01-10T07:06:12.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:06:13.091 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:06:13.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:06:13.405 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.b 2024-01-10T07:06:13.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:06:13.822 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.c 2024-01-10T07:06:13.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-01-10T07:06:14.176 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.d 2024-01-10T07:06:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-01-10T07:06:14.677 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.e 2024-01-10T07:06:14.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-01-10T07:06:14.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:06:14.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:06:14.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:06:14.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:06:14.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:06:14.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:06:14.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-01-10T07:06:14.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-01-10T07:06:14.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1704870434 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870374 -lt 1704870434 ]] 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:14.798 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:15.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:15.031 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.032 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:15.032 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:15.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:15.034 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:15.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870375 -lt 1704870434 ]] 2024-01-10T07:06:15.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:15.035 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:15.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:15.270 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:15.271 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:15.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:15.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:15.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870375 -lt 1704870434 ]] 2024-01-10T07:06:15.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:15.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:15.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:15.507 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:15.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:15.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870375 -lt 1704870434 ]] 2024-01-10T07:06:15.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:15.512 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:15.747 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:15.747 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.748 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:15.748 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:15.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:15.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:15.750 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:15.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870375 -lt 1704870434 ]] 2024-01-10T07:06:15.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:15.751 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:18.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:18.994 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:18.994 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:18.994 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:18.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:18.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:18.997 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:18.998 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870378 -lt 1704870434 ]] 2024-01-10T07:06:18.998 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:18.998 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:22.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:22.241 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:22.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:22.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:22.245 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:22.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870382 -lt 1704870434 ]] 2024-01-10T07:06:22.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:22.246 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:22.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:22.481 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.481 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:22.482 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:22.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:22.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:22.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870382 -lt 1704870434 ]] 2024-01-10T07:06:22.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:22.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:22.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:22.718 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.719 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:22.719 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:22.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:22.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:22.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870382 -lt 1704870434 ]] 2024-01-10T07:06:22.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:22.722 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:22.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:22.956 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:22.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:22.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:22.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:22.960 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:22.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870382 -lt 1704870434 ]] 2024-01-10T07:06:22.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:22.961 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:23.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:23.196 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:23.196 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:23.196 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:23.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-01-10T07:06:23.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-01-10T07:06:23.198 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-01-10T07:06:23.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1704870383 -lt 1704870434 ]] 2024-01-10T07:06:23.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-01-10T07:06:23.199 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-01-10T07:06:30.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-01-10T07:06:30.214 INFO:tasks.workunit.client.0.smithi057.stderr:{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:30.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-01-10T07:06:30.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c4352b7c-e075-4d39-9959-a72f24818694","modified":"2024-01-10T07:05:15.301785Z","created":"2024-01-10T07:05:15.301785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-01-10T07:06:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-01-10T07:06:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-01-10T07:06:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-01-10T07:06:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-01-10T07:06:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-01-10T07:06:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-01-10T07:06:33.473 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-01-10T07:06:33.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-01-10T07:06:33.712 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-01-10T07:06:33.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-01-10T07:06:33.980 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-01-10T07:06:33.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-01-10T07:06:37.161 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-01-10T07:06:37.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-01-10T07:06:43.431 INFO:tasks.workunit.client.0.smithi057.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-01-10T07:06:43.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-01-10T07:06:43.775 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-01-10T07:06:43.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-01-10T07:06:44.027 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.a 2024-01-10T07:06:44.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-01-10T07:06:44.288 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: removing mon.b would break quorum 2024-01-10T07:06:44.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-01-10T07:06:44.568 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: removing mon.c would break quorum 2024-01-10T07:06:44.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-01-10T07:06:47.749 INFO:tasks.workunit.client.0.smithi057.stderr:Error EBUSY: removing mon.d would break quorum 2024-01-10T07:06:47.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-01-10T07:06:48.087 INFO:tasks.workunit.client.0.smithi057.stderr:safe to remove mon.e 2024-01-10T07:06:48.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-01-10T07:06:48.097 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:06:48.097 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:06:48.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:06:48.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:06:48.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:06:48.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:06:48.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:06:48.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:06:48.212 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:06:48.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:06:48.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:06:48.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:06:48.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:06:48.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:06:48.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:06:48.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:06:48.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:06:48.216 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:06:48.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:06:48.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:06:48.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:06:48.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:06:48.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:48.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:06:48.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:06:48.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:06:48.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/ok-to-stop 0 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:06:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:06:48.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:06:48.232 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:06:48.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:06:48.233 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:06:48.234 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:06:48.234 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:06:48.235 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:06:48.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:06:48.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:06:48.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:06:48.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:06:48.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:06:48.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-01-10T07:06:48.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-01-10T07:06:48.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:06:48.240 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:48.240 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64006 2024-01-10T07:06:48.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64006 2024-01-10T07:06:48.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:06:48.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:06:48.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2024-01-10T07:06:48.242 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-10T07:06:48.242 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-10T07:06:48.270 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-01-10T07:06:48.270 DEBUG:teuthology.orchestra.run.smithi057:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-01-10T07:06:48.335 INFO:tasks.workunit.client.0.smithi057.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2024-01-10T07:06:48.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/rados-striper 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-01-10T07:06:48.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-01-10T07:06:48.350 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-01-10T07:06:48.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none ' 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:06:48.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:06:48.352 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:06:48.354 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81752 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:06:48.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81752 2024-01-10T07:06:48.356 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-01-10T07:06:48.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:06:48.765 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:06:48.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:06:49.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:06:49.077 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:06:49.078 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81752/ceph-mon.a.asok 2024-01-10T07:06:49.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:06:49.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81752/ceph-mon.a.asok config get fsid 2024-01-10T07:06:49.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:06:49.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:06:49.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:06:49.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:06:49.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:06:49.186 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81752/ceph-mon.a.asok 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:06:49.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81752/ceph-mon.a.asok config get mon_host 2024-01-10T07:06:49.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-01-10T07:06:49.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:06:49.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:06:49.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a926c47d-9ad9-4bea-8619-0bfd766898be 2024-01-10T07:06:49.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a926c47d-9ad9-4bea-8619-0bfd766898be' 2024-01-10T07:06:49.294 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 a926c47d-9ad9-4bea-8619-0bfd766898be 2024-01-10T07:06:49.294 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:06:49.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJQp5lOQQhEhAADeNnJ5jawnK+xtlGlE/hQA== 2024-01-10T07:06:49.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJQp5lOQQhEhAADeNnJ5jawnK+xtlGlE/hQA=="}' 2024-01-10T07:06:49.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a926c47d-9ad9-4bea-8619-0bfd766898be -i td/rados-striper/0/new.json 2024-01-10T07:06:50.317 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:06:50.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-01-10T07:06:50.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJQp5lOQQhEhAADeNnJ5jawnK+xtlGlE/hQA== --osd-uuid a926c47d-9ad9-4bea-8619-0bfd766898be 2024-01-10T07:06:50.353 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.353+0000 7f1ef6ccb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:06:50.355 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.355+0000 7f1ef6ccb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:06:50.356 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.356+0000 7f1ef6ccb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:06:50.357 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.356+0000 7f1ef6ccb540 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-01-10T07:06:50.357 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.356+0000 7f1ef6ccb540 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-01-10T07:06:50.357 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.356+0000 7f1ef6ccb540 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-01-10T07:06:50.483 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:50.483+0000 7f1ef6ccb540 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-01-10T07:06:56.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-01-10T07:06:56.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:06:56.513 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:06:56.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:06:56.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:06:56.761 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:06:56.761 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:06:56.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:06:56.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:06:56.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:06:56.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:06:56.782 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:56.782+0000 7fc30e462540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:06:56.784 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:56.784+0000 7fc30e462540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:06:56.785 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:56.785+0000 7fc30e462540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:06:56.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:06:56.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:06:56.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:06:56.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:06:56.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:06:56.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:06:56.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:06:56.990 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:06:56.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:06:56.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:06:57.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:06:57.313 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:57.312+0000 7fc30e462540 -1 Falling back to public interface 2024-01-10T07:06:58.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:06:58.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:06:58.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:06:58.215 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:06:58.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:06:58.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:06:58.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:06:59.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:06:59.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:06:59.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:06:59.454 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:06:59.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:06:59.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:06:59.689 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:06:59.689+0000 7fc30e462540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:06:59.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:00.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:00.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:00.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:07:00.693 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:07:00.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:00.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:00.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:01.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:01.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:01.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:07:01.933 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:07:01.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:01.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:02.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:03.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:03.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:03.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:07:03.174 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:07:03.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:03.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:03.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:04.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:04.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:04.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:07:04.415 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:07:04.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:04.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:04.653 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:05.654 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:05.654 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:05.655 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:07:05.655 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:07:05.655 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:05.655 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:05.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:06.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:06.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:06.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:07:06.894 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:07:06.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:06.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:07.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:08.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:08.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:08.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:07:08.132 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:07:08.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:08.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:08.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:09.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:09.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:09.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:07:09.373 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:07:09.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:09.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:09.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:10.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:10.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:10.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:07:10.612 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:07:10.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:10.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:10.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:11.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:11.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:11.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:07:11.853 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:07:11.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:11.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:12.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:13.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:13.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:13.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:07:13.092 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:07:13.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:13.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:13.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:14.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:14.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:14.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:07:14.331 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:07:14.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:14.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:14.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:15.570 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:15.570 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:15.570 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:07:15.571 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:07:15.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:15.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:15.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:16.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:16.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:16.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:07:16.810 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:07:16.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:16.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:17.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:18.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:18.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:18.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:07:18.052 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:07:18.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:18.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:18.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:19.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:19.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:19.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:07:19.288 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:07:19.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:19.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:19.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:20.560 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:20.560 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:20.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-01-10T07:07:20.561 INFO:tasks.workunit.client.0.smithi057.stdout:19 2024-01-10T07:07:20.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:20.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:07:20.808 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/82707,v1:127.0.0.1:6801/82707] [v2:127.0.0.1:6802/82707,v1:127.0.0.1:6803/82707] exists,up a926c47d-9ad9-4bea-8619-0bfd766898be 2024-01-10T07:07:20.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:07:20.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:07:20.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:07:20.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-01-10T07:07:20.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:07:20.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5879b24c-2793-4dd2-a32d-08d48c4004e3 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5879b24c-2793-4dd2-a32d-08d48c4004e3' 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 5879b24c-2793-4dd2-a32d-08d48c4004e3 2024-01-10T07:07:20.815 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:07:20.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoQp5lexwyMRAAMu4S0wx+mINY3NqJqVVdIw== 2024-01-10T07:07:20.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoQp5lexwyMRAAMu4S0wx+mINY3NqJqVVdIw=="}' 2024-01-10T07:07:20.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5879b24c-2793-4dd2-a32d-08d48c4004e3 -i td/rados-striper/1/new.json 2024-01-10T07:07:21.148 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:07:21.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-01-10T07:07:21.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoQp5lexwyMRAAMu4S0wx+mINY3NqJqVVdIw== --osd-uuid 5879b24c-2793-4dd2-a32d-08d48c4004e3 2024-01-10T07:07:21.183 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.182+0000 7f2de645b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:21.220 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.219+0000 7f2de645b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:21.220 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.220+0000 7f2de645b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:21.221 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.221+0000 7f2de645b540 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-01-10T07:07:21.221 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.221+0000 7f2de645b540 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-01-10T07:07:21.221 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.221+0000 7f2de645b540 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-01-10T07:07:21.260 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:21.260+0000 7f2de645b540 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-01-10T07:07:26.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-01-10T07:07:26.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:07:26.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:07:26.988 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:07:26.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:07:27.235 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:07:27.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:07:27.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:07:27.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:07:27.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:07:27.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:07:27.260 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:27.256+0000 7f10300ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:27.260 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:27.258+0000 7f10300ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:27.261 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:27.259+0000 7f10300ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:27.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:07:27.465 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:07:27.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:27.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:27.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:27.788 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:27.788+0000 7f10300ff540 -1 Falling back to public interface 2024-01-10T07:07:28.690 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:28.690 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:28.690 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:07:28.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:07:28.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:28.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:28.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:29.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:29.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:29.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:07:29.929 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:07:29.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:29.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:30.148 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:30.147+0000 7f10300ff540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:07:30.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:31.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:31.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:31.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:07:31.165 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:07:31.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:31.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:31.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:32.404 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:32.404 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:32.404 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:07:32.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:07:32.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:32.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:32.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:33.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:33.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:33.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:07:33.647 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:07:33.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:33.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:33.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:34.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:34.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:34.888 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:07:34.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:07:34.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:34.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:35.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:36.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:36.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:36.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:07:36.130 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:07:36.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:36.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:36.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:37.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:37.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:37.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:07:37.369 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:07:37.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:37.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:37.608 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:38.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:38.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:38.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:07:38.610 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:07:38.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:38.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:38.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:39.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:39.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:39.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:07:39.850 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:07:39.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:39.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:40.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:07:41.088 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:07:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:41.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:42.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:42.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:42.325 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:07:42.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:07:42.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:42.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:42.564 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:43.566 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:43.566 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:43.567 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:07:43.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:07:43.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:43.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:43.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:44.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:44.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:44.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:07:44.805 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:07:44.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:44.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:45.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:46.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:46.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:46.043 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:07:46.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:07:46.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:46.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:46.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:47.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:47.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:47.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:07:47.283 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:07:47.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:47.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:47.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:48.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:48.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:48.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:07:48.520 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:07:48.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:48.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:48.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:49.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:49.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:49.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:07:49.758 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:07:49.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:49.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:07:50.115 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/84534,v1:127.0.0.1:6809/84534] [v2:127.0.0.1:6810/84534,v1:127.0.0.1:6811/84534] exists,up 5879b24c-2793-4dd2-a32d-08d48c4004e3 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:07:50.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:07:50.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1bf39395-2f3b-4b87-b04d-3a52dc889dad 2024-01-10T07:07:50.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1bf39395-2f3b-4b87-b04d-3a52dc889dad' 2024-01-10T07:07:50.122 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 1bf39395-2f3b-4b87-b04d-3a52dc889dad 2024-01-10T07:07:50.122 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:07:50.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGQp5lefflBxAASvM1sU/zBLc/oRMKTWjCYw== 2024-01-10T07:07:50.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGQp5lefflBxAASvM1sU/zBLc/oRMKTWjCYw=="}' 2024-01-10T07:07:50.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1bf39395-2f3b-4b87-b04d-3a52dc889dad -i td/rados-striper/2/new.json 2024-01-10T07:07:50.617 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:07:50.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-01-10T07:07:50.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGQp5lefflBxAASvM1sU/zBLc/oRMKTWjCYw== --osd-uuid 1bf39395-2f3b-4b87-b04d-3a52dc889dad 2024-01-10T07:07:50.652 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.651+0000 7f89cd369540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:50.653 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.653+0000 7f89cd369540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:50.654 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.654+0000 7f89cd369540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:50.655 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.654+0000 7f89cd369540 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-01-10T07:07:50.655 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.655+0000 7f89cd369540 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-01-10T07:07:50.655 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.655+0000 7f89cd369540 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-01-10T07:07:50.700 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:50.700+0000 7f89cd369540 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-01-10T07:07:56.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-01-10T07:07:56.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:07:56.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T07:07:56.050 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T07:07:56.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:07:56.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T07:07:56.298 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T07:07:56.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:07:56.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a4d5b05-2b79-4e68-943e-bd7ef4f1ad40 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.81752/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:07:56.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:07:56.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:07:56.319 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:56.319+0000 7f3c0b234540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:56.321 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:56.321+0000 7f3c0b234540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:56.322 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:56.322+0000 7f3c0b234540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:07:56.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:07:56.528 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:07:56.529 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:56.529 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:07:56.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:56.857 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:56.857+0000 7f3c0b234540 -1 Falling back to public interface 2024-01-10T07:07:57.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:57.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:57.753 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:07:57.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:07:57.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:57.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:07:57.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:07:58.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:07:58.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:07:58.996 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:07:58.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:07:58.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:07:58.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:07:59.207 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:07:59.207+0000 7f3c0b234540 -1 osd.2 0 log_to_monitors true 2024-01-10T07:07:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:00.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:00.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:00.232 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:08:00.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:08:00.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:00.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:00.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:01.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:01.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:01.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:08:01.474 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:08:01.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:01.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:01.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:02.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:02.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:02.718 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:08:02.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:08:02.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:02.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:02.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:03.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:03.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:03.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:08:03.957 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:08:03.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:03.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:04.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:05.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:05.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:05.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:08:05.197 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:08:05.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:05.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:05.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:06.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:06.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:06.439 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:08:06.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:08:06.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:06.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:06.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:07.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:07.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:07.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:08:07.679 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:08:07.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:07.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:07.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:08.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:08.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:08.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:08:08.921 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:08:08.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:08.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:09.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:10.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:10.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:10.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:08:10.161 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:08:10.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:10.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:10.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:11.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:11.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:11.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:08:11.402 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:08:11.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:11.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:11.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:12.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:12.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:12.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:08:12.642 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:08:12.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:12.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:12.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:13.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:13.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:13.883 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:08:13.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:08:13.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:13.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:14.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:15.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:15.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:15.122 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:08:15.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:08:15.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:15.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:15.404 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:16.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:16.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:16.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:08:16.405 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:08:16.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:16.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:16.644 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:17.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:17.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:17.645 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:08:17.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:08:17.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:17.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:17.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:18.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:18.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:18.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:08:18.884 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:08:18.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:18.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:08:19.147 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/86343,v1:127.0.0.1:6817/86343] [v2:127.0.0.1:6818/86343,v1:127.0.0.1:6819/86343] exists,up 1bf39395-2f3b-4b87-b04d-3a52dc889dad 2024-01-10T07:08:19.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:08:19.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:08:19.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:08:19.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-01-10T07:08:19.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:08:19.378 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:08:19.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:08:19.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:08:19.749 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:08:19.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:08:20.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:08:22.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-01-10T07:08:22.086 INFO:tasks.workunit.client.0.smithi057.stderr:1+0 records in 2024-01-10T07:08:22.086 INFO:tasks.workunit.client.0.smithi057.stderr:1+0 records out 2024-01-10T07:08:22.086 INFO:tasks.workunit.client.0.smithi057.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 6.2296e-05 s, 19.8 MB/s 2024-01-10T07:08:22.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-01-10T07:08:22.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-01-10T07:08:22.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-01-10T07:08:22.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-01-10T07:08:22.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-01-10T07:08:22.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-01-10T07:08:22.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-01-10T07:08:22.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-01-10T07:08:22.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-01-10T07:08:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-01-10T07:08:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-01-10T07:08:23.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-01-10T07:08:23.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-01-10T07:08:23.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-01-10T07:08:23.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-01-10T07:08:23.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-01-10T07:08:23.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-01-10T07:08:23.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-01-10T07:08:23.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-01-10T07:08:23.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-01-10T07:08:23.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-01-10T07:08:23.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-01-10T07:08:23.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-01-10T07:08:23.601 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-01-10T07:08:23.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-01-10T07:08:23.651 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-01-10T07:08:23.652 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-01-10T07:08:23.652 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-01-10T07:08:23.653 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-01-10T07:08:23.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-01-10T07:08:23.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-01-10T07:08:23.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-01-10T07:08:23.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-01-10T07:08:23.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-01-10T07:08:23.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-01-10T07:08:23.819 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-01-10T07:08:23.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-01-10T07:08:23.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-01-10T07:08:23.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-01-10T07:08:23.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2024-01-10T07:08:23.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:08:23.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No data available' 2024-01-10T07:08:23.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:08:23.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:08:23.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-01-10T07:08:23.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2024-01-10T07:08:23.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2024-01-10T07:08:23.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No data available' 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:08:23.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-01-10T07:08:23.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2024-01-10T07:08:23.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2024-01-10T07:08:23.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-01-10T07:08:23.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2024-01-10T07:08:23.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-01-10T07:08:24.028 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-01-10T07:08:24.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2024-01-10T07:08:24.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:08:24.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No such file or directory' 2024-01-10T07:08:24.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:08:24.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:08:24.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd --striper stat toyfile 2024-01-10T07:08:24.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2024-01-10T07:08:24.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2024-01-10T07:08:24.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-01-10T07:08:24.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No such file or directory' 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:08:24.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-01-10T07:08:24.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2024-01-10T07:08:24.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2024-01-10T07:08:24.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-01-10T07:08:24.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:08:24.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:08:24.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:08:24.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:08:24.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:08:24.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:08:24.188 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:08:24.188 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:08:24.188 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:08:24.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:08:24.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:08:24.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:08:24.191 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:08:24.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:08:24.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:08:24.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-01-10T07:08:24.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:08:24.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:24.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:08:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81752 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/rados-striper 0 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:08:24.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:08:24.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:08:24.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:08:24.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:08:24.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:08:24.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:08:24.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:08:24.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:08:24.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:08:24.208 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:08:24.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:08:24.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:08:24.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:08:24.211 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:08:24.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:08:24.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-01-10T07:08:24.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-01-10T07:08:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:08:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:24.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81752 2024-01-10T07:08:24.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81752 2024-01-10T07:08:24.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:08:24.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:08:24.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2024-01-10T07:08:24.215 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-10T07:08:24.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-10T07:08:24.283 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-01-10T07:08:24.283 DEBUG:teuthology.orchestra.run.smithi057:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-01-10T07:08:24.349 INFO:tasks.workunit.client.0.smithi057.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2260: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: run_tests: export CEPH_MON=127.0.0.1:7109 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: run_tests: CEPH_MON=127.0.0.1:7109 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: run_tests: export CEPH_ARGS 2024-01-10T07:08:24.356 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: run_tests: uuidgen 2024-01-10T07:08:24.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: run_tests: CEPH_ARGS+=' --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none ' 2024-01-10T07:08:24.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-01-10T07:08:24.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: run_tests: export CEPH_CONF=/dev/null 2024-01-10T07:08:24.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: run_tests: CEPH_CONF=/dev/null 2024-01-10T07:08:24.357 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: run_tests: set 2024-01-10T07:08:24.357 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-01-10T07:08:24.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: run_tests: local 'funcs=test_activate_osd 2024-01-10T07:08:24.359 INFO:tasks.workunit.client.0.smithi057.stderr:test_activate_osd_after_mark_down 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_activate_osd_skip_benchmark 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_destroy_osd 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_display_logs 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_erasure_code_plugin_exists 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_expect_failure 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_flush_pg_stats 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_config 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_is_making_recovery_progress 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_last_scrub_stamp 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_not_primary 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_num_active_clean 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_num_active_or_peered 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_num_pgs 2024-01-10T07:08:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_op_scheduler 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_osd_id_used_by_pgs 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_osds 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_pg 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_primary 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_get_timeout_delays 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_is_clean 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_kill_daemon 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_kill_daemons 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_objectstore_tool 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_pg_scrub 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_repair 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_run_in_background 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_run_mon 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_run_osd 2024-01-10T07:08:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:test_set_config 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_setup 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_teardown 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_background 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_for_clean 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_for_health_ok 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_for_osd 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_for_peered 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_for_scrub 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:test_wait_osd_id_used_by_pgs' 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: run_tests: local dir=td/ceph-helpers 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_activate_osd td/ceph-helpers 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:08:24.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:08:24.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:08:24.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:08:24.364 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:08:24.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:08:24.364 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:08:24.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:08:24.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:08:24.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:08:24.367 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:08:24.367 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:24.367 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:24.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:08:24.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:08:24.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:08:24.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:08:24.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:08:24.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:24.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:24.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:08:24.371 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:08:24.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-01-10T07:08:24.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-01-10T07:08:24.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:08:24.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:08:24.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:08:24.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:08:24.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:08:24.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:08:24.743 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:08:24.743 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:24.743 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:24.744 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:08:24.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:08:24.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:08:24.988 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:08:24.988 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:08:24.988 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:08:24.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:08:25.093 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:08:25.094 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:08:25.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-01-10T07:08:25.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:08:25.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:08:25.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:08:25.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:08:25.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:08:25.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:08:26.090 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:08:26.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:08:26.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:08:26.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:08:26.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a 2024-01-10T07:08:26.123 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a 2024-01-10T07:08:26.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a' 2024-01-10T07:08:26.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:08:26.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqQp5lQLwbCBAAMmfh16TaDMKw5lsx1QitjQ== 2024-01-10T07:08:26.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqQp5lQLwbCBAAMmfh16TaDMKw5lsx1QitjQ=="}' 2024-01-10T07:08:26.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a -i td/ceph-helpers/0/new.json 2024-01-10T07:08:26.538 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:08:26.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:08:26.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqQp5lQLwbCBAAMmfh16TaDMKw5lsx1QitjQ== --osd-uuid 0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a 2024-01-10T07:08:26.576 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.575+0000 7fd3a7dfc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:26.578 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.578+0000 7fd3a7dfc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:26.579 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.579+0000 7fd3a7dfc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:26.579 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.579+0000 7fd3a7dfc540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:08:26.579 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.579+0000 7fd3a7dfc540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:08:26.580 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.579+0000 7fd3a7dfc540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:08:26.655 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:26.655+0000 7fd3a7dfc540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:08:32.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:08:32.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:08:32.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:08:32.209 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:08:32.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:08:32.553 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:08:32.553 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:08:32.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:08:32.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:08:32.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:08:32.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:08:32.576 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:32.576+0000 7f41bf00f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:32.644 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:32.643+0000 7f41bf00f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:32.644 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:32.644+0000 7f41bf00f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:33.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:08:33.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:08:33.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:08:33.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:08:33.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:08:33.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:33.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:08:33.009 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:08:33.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:33.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:33.169 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:33.169+0000 7f41bf00f540 -1 Falling back to public interface 2024-01-10T07:08:33.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:34.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:34.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:34.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:08:34.471 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:08:34.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:34.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:34.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:35.625 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:35.625+0000 7f41bf00f540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:08:35.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:35.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:35.938 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:08:35.938 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:08:35.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:35.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:36.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:37.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:37.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:37.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:08:37.545 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:08:37.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:37.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:38.003 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:39.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:39.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:39.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:08:39.005 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:08:39.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:39.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:39.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:40.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:40.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:40.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:08:40.471 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:08:40.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:40.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:40.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:41.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:41.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:41.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:08:41.941 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:08:41.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:41.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:42.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:43.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:43.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:43.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:08:43.418 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:08:43.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:43.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:43.878 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:44.879 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:44.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:44.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:08:44.880 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:08:44.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:44.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:45.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:46.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:46.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:46.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:08:46.483 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:08:46.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:46.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:46.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:47.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:47.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:47.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:08:47.943 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:08:47.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:47.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:48.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:49.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:49.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:49.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:08:49.906 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:08:49.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:49.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:50.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:51.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:51.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:51.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:08:51.871 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:08:51.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:51.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:52.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:53.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:53.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:53.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:08:53.750 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:08:53.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:53.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:54.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:55.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:55.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:55.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:08:55.624 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:08:55.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:55.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:56.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:08:57.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:08:57.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:57.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:08:57.395 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:08:57.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:57.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:57.956 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/89028,v1:127.0.0.1:6803/89028] [v2:127.0.0.1:6804/89028,v1:127.0.0.1:6805/89028] exists,up 0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-01-10T07:08:57.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T07:08:58.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-01-10T07:08:58.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-01-10T07:08:58.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-01-10T07:08:58.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-01-10T07:08:58.066 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:08:58.067 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:08:58.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:08:58.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:08:58.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:08:58.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:08:58.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-01-10T07:08:58.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:08:58.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:08:58.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:08:58.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-01-10T07:08:58.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:08:58.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-01-10T07:08:58.378 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:08:58.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-01-10T07:08:58.379 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-01-10T07:08:58.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-01-10T07:08:58.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:08:58.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:08:58.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:08:58.399 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:58.399+0000 7f189b34f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:58.402 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:58.401+0000 7f189b34f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:58.403 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:58.403+0000 7f189b34f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:08:58.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:08:58.863 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:08:58.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:08:58.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:08:58.925 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:08:58.925+0000 7f189b34f540 -1 Falling back to public interface 2024-01-10T07:08:59.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:00.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:00.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:00.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:09:00.312 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:09:00.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:00.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:00.781 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:01.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:01.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:01.782 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:09:01.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:09:01.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:01.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:01.847 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:01.847+0000 7f189b34f540 -1 osd.0 6 log_to_monitors true 2024-01-10T07:09:02.253 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/89945,v1:127.0.0.1:6803/89945] [v2:127.0.0.1:6804/89945,v1:127.0.0.1:6805/89945] exists,up 0cdfb08a-1b2b-4b5a-be5a-ad37753cf10a 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:02.254 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:02.255 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:02.255 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:09:02.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-01-10T07:09:02.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T07:09:02.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-01-10T07:09:02.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-01-10T07:09:02.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-01-10T07:09:02.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:09:02.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:09:02.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:09:02.364 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:09:02.364 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:09:02.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:09:02.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:09:02.364 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:09:02.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:09:02.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:09:02.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:09:02.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:09:02.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:09:02.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:09:02.471 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:09:02.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:02.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:09:02.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:09:02.473 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:02.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:09:02.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:09:02.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:09:02.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:09:02.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:02.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:02.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:09:02.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:09:02.485 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:09:02.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:09:02.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:09:02.486 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:09:02.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:02.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:09:02.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:09:02.487 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:02.488 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:09:02.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:09:02.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:09:02.490 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:09:02.490 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:02.490 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:02.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:09:02.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:09:02.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:09:02.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:09:02.492 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:09:02.492 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:02.492 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:02.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:09:02.494 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:09:02.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:09:02.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:09:02.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:09:02.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-01-10T07:09:02.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:09:02.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:09:02.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:09:02.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:09:02.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:09:02.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:02.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:09:03.110 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:09:03.111 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:09:03.111 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:09:03.111 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:09:03.111 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:09:03.111 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:09:03.112 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:09:03.112 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:09:03.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:09:03.219 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:09:03.219 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:09:03.219 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:09:03.219 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:09:03.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:09:03.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:04.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:09:04.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:09:04.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:09:04.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:09:04.245 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:09:04.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=462a072d-bcfc-4a8e-9fdb-e94a5c0da520 2024-01-10T07:09:04.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 462a072d-bcfc-4a8e-9fdb-e94a5c0da520' 2024-01-10T07:09:04.246 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 462a072d-bcfc-4a8e-9fdb-e94a5c0da520 2024-01-10T07:09:04.247 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:09:04.259 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQQp5lS1tzDxAAmcLTs2Ig2PhtTM2G7xy1mw== 2024-01-10T07:09:04.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQQp5lS1tzDxAAmcLTs2Ig2PhtTM2G7xy1mw=="}' 2024-01-10T07:09:04.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 462a072d-bcfc-4a8e-9fdb-e94a5c0da520 -i td/ceph-helpers/0/new.json 2024-01-10T07:09:04.661 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:04.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:09:04.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQQp5lS1tzDxAAmcLTs2Ig2PhtTM2G7xy1mw== --osd-uuid 462a072d-bcfc-4a8e-9fdb-e94a5c0da520 2024-01-10T07:09:04.698 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.698+0000 7fdf64fe9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:04.700 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.700+0000 7fdf64fe9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:04.701 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.701+0000 7fdf64fe9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:04.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.701+0000 7fdf64fe9540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:09:04.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.701+0000 7fdf64fe9540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:09:04.702 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.702+0000 7fdf64fe9540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:09:04.774 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:04.774+0000 7fdf64fe9540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:09:10.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:09:10.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:09:10.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:09:10.089 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:09:10.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:09:10.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:09:10.332 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:09:10.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:09:10.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:09:10.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:09:10.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:09:10.355 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:10.355+0000 7f8b52ba8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:10.358 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:10.358+0000 7f8b52ba8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:10.359 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:10.359+0000 7f8b52ba8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:10.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:09:10.603 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:10.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:10.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:10.878 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:10.878+0000 7f8b52ba8540 -1 Falling back to public interface 2024-01-10T07:09:11.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:12.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:12.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:12.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:09:12.139 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:09:12.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:12.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:12.621 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:13.295 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:13.294+0000 7f8b52ba8540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:09:13.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:13.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:13.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:09:13.623 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:09:13.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:13.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:14.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:15.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:15.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:15.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:09:15.083 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:09:15.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:15.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:15.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:16.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:16.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:16.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:09:16.550 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:09:16.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:16.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:17.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:18.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:18.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:18.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:09:18.062 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:09:18.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:18.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:18.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:19.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:19.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:19.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:09:19.530 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:09:19.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:19.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:20.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:21.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:21.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:21.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:09:21.029 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:09:21.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:21.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:21.488 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:22.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:22.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:22.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:09:22.490 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:09:22.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:22.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:23.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:24.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:24.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:24.008 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:09:24.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:09:24.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:24.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:24.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:25.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:25.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:25.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:09:25.754 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:09:25.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:25.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:26.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:27.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:27.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:27.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:09:27.758 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:09:27.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:27.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:28.608 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:29.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:29.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:29.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:09:29.610 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:09:29.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:29.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:30.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:31.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:31.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:31.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:09:31.590 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:09:31.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:31.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:32.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:33.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:33.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:33.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:09:33.593 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:09:33.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:33.594 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:34.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:35.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:35.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:35.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:09:35.179 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:09:35.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:35.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:35.657 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91581,v1:127.0.0.1:6803/91581] [v2:127.0.0.1:6804/91581,v1:127.0.0.1:6805/91581] exists,up 462a072d-bcfc-4a8e-9fdb-e94a5c0da520 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:09:35.659 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-01-10T07:09:35.660 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T07:09:35.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-01-10T07:09:35.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-01-10T07:09:35.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-01-10T07:09:35.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-01-10T07:09:35.767 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:09:35.768 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:09:35.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:09:35.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:09:35.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:09:37.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:09:37.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-01-10T07:09:37.551 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already down. 2024-01-10T07:09:37.561 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-01-10T07:09:37.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-01-10T07:09:37.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:09:37.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:09:37.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:09:37.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:37.562 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:37.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:09:37.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:37.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-01-10T07:09:38.062 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/91581,v1:127.0.0.1:6803/91581] [v2:127.0.0.1:6804/91581,v1:127.0.0.1:6805/91581] exists 462a072d-bcfc-4a8e-9fdb-e94a5c0da520 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:09:38.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:38.067 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:09:38.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-01-10T07:09:38.069 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:09:38.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-01-10T07:09:38.069 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-01-10T07:09:38.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-01-10T07:09:38.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:09:38.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:09:38.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:09:38.088 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:38.087+0000 7fd0fc348540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:38.090 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:38.089+0000 7fd0fc348540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:38.090 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:38.090+0000 7fd0fc348540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:38.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-01-10T07:09:38.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:09:38.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:09:38.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:09:38.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:09:38.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:38.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:09:38.542 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:38.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:38.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:38.611 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:38.609+0000 7fd0fc348540 -1 Falling back to public interface 2024-01-10T07:09:39.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:40.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:40.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:40.075 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:09:40.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:09:40.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:40.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:40.607 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:41.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:41.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:41.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:09:41.609 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:09:41.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:41.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:41.625 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:41.624+0000 7fd0fc348540 -1 osd.0 6 log_to_monitors true 2024-01-10T07:09:42.116 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/92600,v1:127.0.0.1:6803/92600] [v2:127.0.0.1:6804/92600,v1:127.0.0.1:6805/92600] exists,up 462a072d-bcfc-4a8e-9fdb-e94a5c0da520 2024-01-10T07:09:42.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:09:42.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:09:42.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-01-10T07:09:42.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:09:42.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:09:42.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:09:42.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:09:42.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:09:42.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:09:42.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:09:42.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:09:42.333 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:09:42.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:42.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:09:42.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:09:42.335 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:42.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:09:42.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:09:42.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:09:42.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:09:42.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:42.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:42.342 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:09:42.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:09:42.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:09:42.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:09:42.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:09:42.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:09:42.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:09:42.346 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:09:42.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:09:42.346 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:09:42.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:09:42.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:09:42.347 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:09:42.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:42.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:09:42.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:09:42.350 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:09:42.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:09:42.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:09:42.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:09:42.352 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:09:42.352 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:42.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:42.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:09:42.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:09:42.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:09:42.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:09:42.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:09:42.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:42.355 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:42.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:09:42.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:42.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:09:42.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:09:42.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:09:42.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:09:42.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:09:42.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:09:42.979 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:09:42.979 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:09:42.980 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:09:43.086 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:09:43.087 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:09:43.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-01-10T07:09:43.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:09:43.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:09:43.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:09:43.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:09:43.188 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:09:43.188 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:09:44.095 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:09:44.095 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:09:44.096 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:09:44.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:09:44.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:09:44.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:09:44.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7729161d-a13e-4a07-b91f-21601afa45cf 2024-01-10T07:09:44.128 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 7729161d-a13e-4a07-b91f-21601afa45cf 2024-01-10T07:09:44.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7729161d-a13e-4a07-b91f-21601afa45cf' 2024-01-10T07:09:44.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:09:44.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4Qp5l+kVrCBAADndeh75gFQE0ph7OoO+9mw== 2024-01-10T07:09:44.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4Qp5l+kVrCBAADndeh75gFQE0ph7OoO+9mw=="}' 2024-01-10T07:09:44.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7729161d-a13e-4a07-b91f-21601afa45cf -i td/ceph-helpers/0/new.json 2024-01-10T07:09:44.545 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:44.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:09:44.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQC4Qp5l+kVrCBAADndeh75gFQE0ph7OoO+9mw== --osd-uuid 7729161d-a13e-4a07-b91f-21601afa45cf 2024-01-10T07:09:44.583 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.583+0000 7f4bef06a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:44.585 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.585+0000 7f4bef06a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:44.587 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.586+0000 7f4bef06a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:44.587 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.586+0000 7f4bef06a540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:09:44.587 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.586+0000 7f4bef06a540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:09:44.587 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.586+0000 7f4bef06a540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:09:44.660 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:44.659+0000 7f4bef06a540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:09:50.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:09:50.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:09:50.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:09:50.196 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:09:50.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:09:50.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:09:50.504 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:09:50.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-01-10T07:09:50.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:09:50.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:09:50.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:09:50.530 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:50.530+0000 7ff0d0a84540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:50.538 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:50.538+0000 7ff0d0a84540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:50.539 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:50.539+0000 7ff0d0a84540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:50.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:09:50.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:09:50.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:09:50.996 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:09:50.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:09:50.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:50.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:09:50.997 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:50.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:50.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:51.072 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:51.072+0000 7ff0d0a84540 -1 Falling back to public interface 2024-01-10T07:09:51.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:52.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:52.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:52.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:09:52.498 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:09:52.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:52.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:53.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:53.569 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:53.568+0000 7ff0d0a84540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:09:54.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:54.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:54.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:09:54.006 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:09:54.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:54.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:54.537 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/94294,v1:127.0.0.1:6803/94294] [v2:127.0.0.1:6804/94294,v1:127.0.0.1:6805/94294] exists,up 7729161d-a13e-4a07-b91f-21601afa45cf 2024-01-10T07:09:54.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:09:54.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:09:54.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-01-10T07:09:54.539 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-01-10T07:09:54.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-01-10T07:09:54.647 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-01-10T07:09:54.647 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-01-10T07:09:54.648 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-01-10T07:09:54.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-01-10T07:09:54.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-01-10T07:09:54.756 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:09:54.756 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:09:54.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:09:54.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:09:54.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:09:55.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:09:55.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-01-10T07:09:55.574 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already down. 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:55.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:09:55.585 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:55.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:55.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-01-10T07:09:56.093 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/94294,v1:127.0.0.1:6803/94294] [v2:127.0.0.1:6804/94294,v1:127.0.0.1:6805/94294] exists 7729161d-a13e-4a07-b91f-21601afa45cf 2024-01-10T07:09:56.096 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:09:56.096 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:09:56.096 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:09:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:09:56.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:09:56.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:09:56.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-01-10T07:09:56.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:09:56.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-01-10T07:09:56.099 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:09:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-01-10T07:09:56.100 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-01-10T07:09:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-01-10T07:09:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:09:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:09:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:09:56.120 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:56.120+0000 7fda7d120540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:56.122 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:56.122+0000 7fda7d120540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:56.123 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:56.123+0000 7fda7d120540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:09:56.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-01-10T07:09:56.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:09:56.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:56.605 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:56.646 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:56.646+0000 7fda7d120540 -1 Falling back to public interface 2024-01-10T07:09:57.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:58.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:58.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:58.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:09:58.114 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:09:58.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:58.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:09:58.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:09:59.430 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:09:59.429+0000 7fda7d120540 -1 osd.0 6 log_to_monitors true 2024-01-10T07:09:59.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:09:59.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:09:59.589 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:09:59.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:09:59.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:09:59.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:00.055 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/94926,v1:127.0.0.1:6803/94926] [v2:127.0.0.1:6804/94926,v1:127.0.0.1:6805/94926] exists,up 7729161d-a13e-4a07-b91f-21601afa45cf 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:00.057 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:10:00.058 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-01-10T07:10:00.058 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-01-10T07:10:00.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:00.165 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:10:00.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-01-10T07:10:00.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-01-10T07:10:00.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-01-10T07:10:00.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-01-10T07:10:00.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-01-10T07:10:00.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:00.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:00.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:00.379 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:00.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:00.380 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:00.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:00.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:00.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:00.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:00.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:00.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:00.383 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:00.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:00.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:00.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:00.388 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:00.388 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:00.388 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:00.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_destroy_osd td/ceph-helpers 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:00.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:00.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:00.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:00.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:00.392 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:00.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:00.393 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:00.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:00.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:00.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:00.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:00.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:00.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:00.396 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:00.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:00.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:00.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:00.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:00.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:00.398 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:00.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:00.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:00.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:00.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:10:00.400 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:10:00.400 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:00.401 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:00.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:10:00.402 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:10:00.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:10:00.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:10:00.756 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:00.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:10:01.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:10:01.018 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:01.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:01.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:01.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:01.124 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:01.125 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:10:01.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-01-10T07:10:01.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:10:01.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:10:01.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:10:01.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:10:01.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:10:01.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:10:02.127 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:10:02.127 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:02.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:02.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:02.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:02.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:02.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:02.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:10:02.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:10:02.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-01-10T07:10:02.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:10:02.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:10:02.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:02.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:10:02.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:10:02.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:10:02.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:10:02.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e830009-d639-439a-9c60-920f25174fdd 2024-01-10T07:10:02.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0e830009-d639-439a-9c60-920f25174fdd' 2024-01-10T07:10:02.160 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 0e830009-d639-439a-9c60-920f25174fdd 2024-01-10T07:10:02.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:10:02.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKQp5lLs1cChAAYO9YNKWUxrRyrs77l7PiMg== 2024-01-10T07:10:02.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKQp5lLs1cChAAYO9YNKWUxrRyrs77l7PiMg=="}' 2024-01-10T07:10:02.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e830009-d639-439a-9c60-920f25174fdd -i td/ceph-helpers/0/new.json 2024-01-10T07:10:02.618 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:10:02.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:10:02.630 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKQp5lLs1cChAAYO9YNKWUxrRyrs77l7PiMg== --osd-uuid 0e830009-d639-439a-9c60-920f25174fdd 2024-01-10T07:10:02.655 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.654+0000 7f2098738540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:02.657 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.657+0000 7f2098738540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:02.658 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.658+0000 7f2098738540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:02.658 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.658+0000 7f2098738540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:10:02.659 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.658+0000 7f2098738540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:10:02.659 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.658+0000 7f2098738540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:10:02.733 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:02.733+0000 7f2098738540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:10:08.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:10:08.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:10:08.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:10:08.047 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:10:08.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:10:08.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:10:08.291 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:10:08.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:10:08.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:10:08.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:10:08.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:10:08.315 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:08.315+0000 7f1c018ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:08.328 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:08.328+0000 7f1c018ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:08.329 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:08.329+0000 7f1c018ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:08.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:10:08.558 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:10:08.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:08.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:08.871+0000 7f1c018ef540 -1 Falling back to public interface 2024-01-10T07:10:09.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:10.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:10.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:10.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:10:10.053 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:10:10.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:10.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:10.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:11.258 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:11.258+0000 7f1c018ef540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:10:11.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:11.537 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:11.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:10:11.538 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:10:11.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:11.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:12.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:13.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:13.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:13.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:10:13.103 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:10:13.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:13.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:13.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:14.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:14.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:14.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:10:14.577 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:10:14.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:14.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:15.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:16.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:16.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:16.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:10:16.052 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:10:16.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:16.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:16.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:17.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:17.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:17.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:10:17.520 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:10:17.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:17.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:18.059 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:19.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:19.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:19.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:10:19.061 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:10:19.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:19.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:19.522 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:20.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:20.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:20.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:10:20.524 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:10:20.525 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:20.525 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:21.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:22.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:22.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:22.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:10:22.041 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:10:22.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:22.042 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:22.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:23.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:23.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:23.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:10:23.503 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:10:23.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:23.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:24.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:25.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:25.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:25.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:10:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:10:25.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:25.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:25.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:26.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:26.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:26.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:10:26.959 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:10:26.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:26.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:27.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:28.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:28.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:28.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:10:28.979 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:10:28.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:28.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:29.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:30.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:30.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:30.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:10:30.478 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:10:30.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:30.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:31.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:32.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:32.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:32.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:10:32.328 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:10:32.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:32.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:32.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:33.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:33.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:33.811 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:10:33.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:10:33.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:33.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:34.291 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/96575,v1:127.0.0.1:6803/96575] [v2:127.0.0.1:6804/96575,v1:127.0.0.1:6805/96575] exists,up 0e830009-d639-439a-9c60-920f25174fdd 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-01-10T07:10:34.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-01-10T07:10:34.941 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already out. 2024-01-10T07:10:34.950 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-01-10T07:10:34.951 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:34.951 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:34.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:34.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:34.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:35.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:35.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-01-10T07:10:35.718 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already down. 2024-01-10T07:10:35.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-01-10T07:10:36.256 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 does not exist 2024-01-10T07:10:36.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-01-10T07:10:36.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-01-10T07:10:36.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:36.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-01-10T07:10:36.266 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:36.267 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:36.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:36.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:36.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:36.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:36.268 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:36.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:36.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:36.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:36.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:36.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:36.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:36.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:36.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:36.272 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:36.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:36.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:36.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-01-10T07:10:36.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:36.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:36.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:36.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:36.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:36.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:36.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-01-10T07:10:36.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-01-10T07:10:36.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:36.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:36.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:36.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:36.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:36.882 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:36.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:36.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:36.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:36.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:36.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:36.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:36.885 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:36.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:36.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:36.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:36.890 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:36.890 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:36.890 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:36.890 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_display_logs td/ceph-helpers 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: test_display_logs: local dir=td/ceph-helpers 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: test_display_logs: setup td/ceph-helpers 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:36.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:36.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:36.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:36.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:36.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:36.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:36.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:36.895 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:36.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:36.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:36.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:36.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:36.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:36.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:36.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:36.900 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:36.900 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:36.900 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:36.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:36.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:36.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:36.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:10:36.902 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:10:36.902 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:36.902 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:36.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:10:36.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:10:36.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_display_logs: run_mon td/ceph-helpers a 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:10:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:10:37.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:10:37.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:37.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:37.259 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:37.259 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:37.259 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:37.259 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:37.259 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:10:37.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:10:37.502 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:37.503 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:37.504 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:37.504 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:37.504 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:37.504 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:37.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:37.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:37.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:37.610 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:10:37.710 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2038: test_display_logs: kill_daemons td/ceph-helpers 2024-01-10T07:10:37.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:37.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:37.711 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:37.711 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:37.711 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:39.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:39.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_display_logs: display_logs td/ceph-helpers 2024-01-10T07:10:39.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: display_logs: local dir=td/ceph-helpers 2024-01-10T07:10:39.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-01-10T07:10:39.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: display_logs: read file 2024-01-10T07:10:39.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-01-10T07:10:39.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: display_logs: cat td/ceph-helpers/mon.a.log 2024-01-10T07:10:39.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: display_logs: read file 2024-01-10T07:10:39.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: test_display_logs: teardown td/ceph-helpers 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:39.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:39.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:39.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:39.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:39.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:39.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:39.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:39.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:39.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:39.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:39.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:39.030 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:39.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:39.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:39.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:39.034 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:39.034 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:39.034 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:39.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:39.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:39.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:39.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:39.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:39.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:39.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:39.038 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:39.039 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:39.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:39.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:39.041 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:39.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:39.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:39.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:39.043 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:39.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:39.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:39.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:39.046 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:39.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:39.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:39.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:39.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:39.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:39.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:10:39.049 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:10:39.049 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:39.049 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:39.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:10:39.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:10:39.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:10:39.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:39.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:39.423 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:39.424 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:39.424 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:39.424 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:39.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:10:39.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:10:39.684 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:10:39.684 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:39.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:39.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:39.686 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:39.799 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:39.800 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:39.800 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:10:39.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-01-10T07:10:39.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:10:39.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:10:39.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:10:39.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:10:39.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:10:39.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:10:40.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:40.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:10:40.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:10:40.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-01-10T07:10:40.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: erasure_code_plugin_exists: local plugin=jerasure 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: erasure_code_plugin_exists: local status 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: erasure_code_plugin_exists: local grepstr 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: erasure_code_plugin_exists: local s 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: case `uname` in 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: uname 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-01-10T07:10:40.837 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-01-10T07:10:41.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: s= 2024-01-10T07:10:41.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: erasure_code_plugin_exists: local status=0 2024-01-10T07:10:41.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-01-10T07:10:41.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-01-10T07:10:41.661 INFO:tasks.workunit.client.0.smithi057.stderr:erasure-code-profile TESTPROFILE does not exist 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: erasure_code_plugin_exists: return 0 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: erasure_code_plugin_exists: local plugin=FAKE 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: erasure_code_plugin_exists: local status 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: erasure_code_plugin_exists: local grepstr 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: erasure_code_plugin_exists: local s 2024-01-10T07:10:41.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: case `uname` in 2024-01-10T07:10:41.673 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: uname 2024-01-10T07:10:41.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-01-10T07:10:41.673 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-01-10T07:10:41.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-01-10T07:10:41.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: erasure_code_plugin_exists: local status=5 2024-01-10T07:10:41.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-01-10T07:10:41.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-01-10T07:10:41.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: erasure_code_plugin_exists: return 5 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2012: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:41.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:41.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:41.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:42.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:42.040 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:42.040 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:42.040 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:42.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:42.041 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:42.041 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:42.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:42.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:42.044 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:42.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:42.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:42.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:42.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.047 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:42.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:42.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:42.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:10:42.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_expect_failure td/ceph-helpers 2024-01-10T07:10:42.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_expect_failure: local dir=td/ceph-helpers 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: test_expect_failure: setup td/ceph-helpers 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:42.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:42.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:42.051 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:42.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:42.051 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:42.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:42.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:42.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:42.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:42.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:42.055 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:42.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:42.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:42.057 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:42.057 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.057 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:42.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:42.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:42.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:10:42.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:10:42.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/ceph-helpers 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local expected=FAIL 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:10:42.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/ceph-helpers 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local expected=FAIL 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:10:42.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-01-10T07:10:42.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: expect_failure: success=true 2024-01-10T07:10:42.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: true 2024-01-10T07:10:42.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: expect_failure: cat td/ceph-helpers/out 2024-01-10T07:10:42.065 INFO:tasks.workunit.client.0.smithi057.stderr:FAIL 2024-01-10T07:10:42.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: expect_failure: return 1 2024-01-10T07:10:42.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-01-10T07:10:42.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/ceph-helpers 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local expected=FAIL 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-01-10T07:10:42.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2024-01-10T07:10:42.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2024-01-10T07:10:42.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-01-10T07:10:42.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: expect_failure: cat td/ceph-helpers/out 2024-01-10T07:10:42.069 INFO:tasks.workunit.client.0.smithi057.stderr:UNEXPECTED 2024-01-10T07:10:42.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: expect_failure: return 1 2024-01-10T07:10:42.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1924: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-01-10T07:10:42.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: test_expect_failure: teardown td/ceph-helpers 2024-01-10T07:10:42.070 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:42.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:42.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:42.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:42.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:42.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:42.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:42.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:42.074 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:42.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:42.076 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:42.076 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.077 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:42.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:42.078 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:42.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:42.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_flush_pg_stats td/ceph-helpers 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: test_flush_pg_stats: local dir=td/ceph-helpers 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: test_flush_pg_stats: setup td/ceph-helpers 2024-01-10T07:10:42.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:10:42.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:10:42.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:10:42.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:10:42.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:10:42.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:10:42.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:10:42.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:10:42.084 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:10:42.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:10:42.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:10:42.086 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:10:42.087 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:10:42.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:10:42.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:10:42.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:10:42.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:10:42.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:10:42.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:10:42.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:10:42.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:10:42.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:10:42.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:10:42.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:10:42.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:10:42.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:10:42.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:10:42.441 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:10:42.441 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:42.442 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:42.442 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:42.442 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.442 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.442 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:42.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:10:42.710 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:42.711 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:42.712 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:42.712 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.712 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.713 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:42.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:42.713 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:42.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:10:42.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:10:42.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:10:42.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-01-10T07:10:42.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:10:42.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:10:42.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:10:42.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:10:42.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:10:42.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:10:43.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:10:43.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:10:43.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:10:43.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:10:43.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:10:43.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:10:43.844 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:10:43.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d01ccb92-104a-43c6-80a9-489a45843540 2024-01-10T07:10:43.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d01ccb92-104a-43c6-80a9-489a45843540' 2024-01-10T07:10:43.845 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 d01ccb92-104a-43c6-80a9-489a45843540 2024-01-10T07:10:43.845 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:10:43.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzQp5lfVkjMxAAwLuRbPYwSVNb58b4Qngg+w== 2024-01-10T07:10:43.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzQp5lfVkjMxAAwLuRbPYwSVNb58b4Qngg+w=="}' 2024-01-10T07:10:43.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d01ccb92-104a-43c6-80a9-489a45843540 -i td/ceph-helpers/0/new.json 2024-01-10T07:10:44.277 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:10:44.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:10:44.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzQp5lfVkjMxAAwLuRbPYwSVNb58b4Qngg+w== --osd-uuid d01ccb92-104a-43c6-80a9-489a45843540 2024-01-10T07:10:44.315 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.315+0000 7fa6c3ae5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:44.317 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.317+0000 7fa6c3ae5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.318+0000 7fa6c3ae5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.318+0000 7fa6c3ae5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:10:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.318+0000 7fa6c3ae5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:10:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.318+0000 7fa6c3ae5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:10:44.394 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:44.393+0000 7fa6c3ae5540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:10:49.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:10:49.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:10:49.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:10:49.693 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:10:49.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:10:49.947 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:10:49.947 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:10:49.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:10:49.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:10:49.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:10:49.949 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:10:49.971 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:49.971+0000 7f30689ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:49.988 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:49.988+0000 7f30689ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:49.989 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:49.989+0000 7f30689ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:10:50.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:10:50.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:50.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:50.520 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:50.520+0000 7f30689ae540 -1 Falling back to public interface 2024-01-10T07:10:50.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:51.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:51.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:51.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:10:51.730 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:10:51.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:51.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:52.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:52.943 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:10:52.943+0000 7f30689ae540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:10:53.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:53.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:53.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:10:53.206 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:10:53.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:53.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:53.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:54.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:54.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:54.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:10:54.668 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:10:54.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:54.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:55.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:56.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:56.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:56.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:10:56.129 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:10:56.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:56.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:56.631 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:57.632 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:57.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:57.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:10:57.633 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:10:57.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:57.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:58.111 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:10:59.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:10:59.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:10:59.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:10:59.113 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:10:59.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:10:59.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:10:59.602 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:00.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:00.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:00.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:11:00.603 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:11:00.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:00.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:01.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:02.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:02.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:02.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:11:02.067 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:11:02.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:02.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:02.538 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:03.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:03.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:03.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:11:03.540 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:11:03.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:03.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:04.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:05.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:05.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:05.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:11:05.211 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:11:05.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:05.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:05.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:06.873 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:06.873 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:06.874 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:11:06.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:11:06.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:06.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:07.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:08.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:08.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:08.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:11:08.843 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:11:08.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:08.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:09.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:10.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:10.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:10.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:11:10.502 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:11:10.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:10.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:11.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:12.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:12.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:12.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:11:12.010 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:11:12.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:12.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:13.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:14.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:14.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:14.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:11:14.026 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:11:14.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:14.027 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:14.512 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/99182,v1:127.0.0.1:6803/99182] [v2:127.0.0.1:6804/99182,v1:127.0.0.1:6805/99182] exists,up d01ccb92-104a-43c6-80a9-489a45843540 2024-01-10T07:11:14.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:11:14.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:11:14.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:11:14.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_flush_pg_stats: create_rbd_pool 2024-01-10T07:11:14.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:11:15.003 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:11:15.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:11:15.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:11:15.865 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:11:15.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:11:16.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:11:17.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2164: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-01-10T07:11:18.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_flush_pg_stats: flush_pg_stats 2024-01-10T07:11:18.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:11:18.030 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:11:18.517 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:11:18.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:11:18.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:11:18.518 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:11:18.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836488 2024-01-10T07:11:18.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836488 2024-01-10T07:11:18.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836488' 2024-01-10T07:11:18.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:11:18.665 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836488 2024-01-10T07:11:18.665 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:11:18.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:11:18.666 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836488 2024-01-10T07:11:18.666 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:11:18.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836488 2024-01-10T07:11:18.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-01-10T07:11:18.667 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836488 2024-01-10T07:11:18.667 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:11:19.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836486 -lt 21474836488 2024-01-10T07:11:19.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:11:20.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:11:20.163 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:11:20.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836488 2024-01-10T07:11:20.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-01-10T07:11:20.638 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_flush_pg_stats: ceph df detail --format=json 2024-01-10T07:11:20.638 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-01-10T07:11:21.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_flush_pg_stats: stored=6012 2024-01-10T07:11:21.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_flush_pg_stats: ceph df detail --format=json 2024-01-10T07:11:21.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-01-10T07:11:21.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_flush_pg_stats: stored_raw=6012 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: test_flush_pg_stats: test 6012 -gt 0 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: test_flush_pg_stats: test 6012 == 6012 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: test_flush_pg_stats: teardown td/ceph-helpers 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:11:21.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:11:22.003 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:11:22.003 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:11:22.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:11:22.004 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:11:22.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:11:22.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:11:22.005 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:11:22.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:22.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:11:22.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:11:22.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:22.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:11:22.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:11:22.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:11:22.015 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:11:22.015 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.015 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_config td/ceph-helpers 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-01-10T07:11:22.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:11:22.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:11:22.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:11:22.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:11:22.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:11:22.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:11:22.021 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:11:22.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:22.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:11:22.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:11:22.023 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:22.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:11:22.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:11:22.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:11:22.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:11:22.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:11:22.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:11:22.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:11:22.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:11:22.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:11:22.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.028 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.028 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:11:22.029 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:11:22.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:11:22.409 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:11:22.409 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:22.409 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:22.409 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:22.409 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.410 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.410 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:22.410 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:11:22.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:11:22.670 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:11:22.670 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:22.670 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:22.670 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:11:22.671 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:11:22.671 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:22.671 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:22.671 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:22.671 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:22.672 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.672 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.672 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:22.672 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:22.672 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:22.779 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.780 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.780 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:22.780 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:22.780 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:11:22.880 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:22.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:22.882 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get osd_pool_default_size 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:11:22.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:11:22.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:11:23.468 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:11:23.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:11:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-01-10T07:11:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:11:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:11:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:11:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:11:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:11:23.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:11:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e165f218-f5c1-4171-9896-f46301b94e3a 2024-01-10T07:11:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e165f218-f5c1-4171-9896-f46301b94e3a' 2024-01-10T07:11:23.500 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 e165f218-f5c1-4171-9896-f46301b94e3a 2024-01-10T07:11:23.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:11:23.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbQ55luZOcHhAAPtG+3bzYi4ZwuMOvJ1GyzA== 2024-01-10T07:11:23.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbQ55luZOcHhAAPtG+3bzYi4ZwuMOvJ1GyzA=="}' 2024-01-10T07:11:23.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e165f218-f5c1-4171-9896-f46301b94e3a -i td/ceph-helpers/0/new.json 2024-01-10T07:11:23.912 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:11:23.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:11:23.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQAbQ55luZOcHhAAPtG+3bzYi4ZwuMOvJ1GyzA== --osd-uuid e165f218-f5c1-4171-9896-f46301b94e3a 2024-01-10T07:11:23.949 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:23.949+0000 7f76bdb75540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:23.951 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:23.951+0000 7f76bdb75540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:23.952 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:23.952+0000 7f76bdb75540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:23.953 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:23.952+0000 7f76bdb75540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:11:23.953 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:23.953+0000 7f76bdb75540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:11:23.953 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:23.953+0000 7f76bdb75540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:11:24.002 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:24.002+0000 7f76bdb75540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:11:29.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:11:29.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:11:29.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:11:29.288 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:11:29.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:11:29.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:11:29.531 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:11:29.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-01-10T07:11:29.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:11:29.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:11:29.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:11:29.556 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:29.555+0000 7fd1e3d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:29.558 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:29.557+0000 7fd1e3d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:29.559 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:29.558+0000 7fd1e3d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:29.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:11:29.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:11:29.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:29.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:30.075 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:30.074+0000 7fd1e3d55540 -1 Falling back to public interface 2024-01-10T07:11:30.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:31.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:31.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:31.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:11:31.315 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:11:31.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:31.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:31.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:32.452 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:32.450+0000 7fd1e3d55540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:11:32.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:32.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:32.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:11:32.799 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:11:32.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:32.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:33.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:34.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:11:34.306 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:11:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:34.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:34.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:35.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:35.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:35.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:11:35.772 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:11:35.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:35.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:36.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:37.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:37.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:37.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:11:37.275 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:11:37.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:37.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:37.735 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:38.736 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:38.736 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:38.736 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:11:38.737 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:11:38.737 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:38.737 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:39.234 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:40.235 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:40.235 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:40.235 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:11:40.236 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:11:40.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:40.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:40.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:41.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:41.700 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:41.701 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:11:41.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:11:41.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:41.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:42.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:43.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:43.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:43.254 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:11:43.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:11:43.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:43.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:43.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:44.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:44.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:44.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:11:44.943 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:11:44.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:44.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:46.028 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:47.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:47.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:47.029 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:11:47.029 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:11:47.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:47.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:48.006 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:49.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:49.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:49.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:11:49.009 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:11:49.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:49.010 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:49.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:50.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:50.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:50.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:11:50.830 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:11:50.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:50.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:51.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:52.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:52.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:52.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:11:52.816 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:11:52.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:52.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:53.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:11:54.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:11:54.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:11:54.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:11:54.576 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:11:54.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:11:54.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:11:55.057 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101458,v1:127.0.0.1:6803/101458] [v2:127.0.0.1:6804/101458,v1:127.0.0.1:6805/101458] exists,up e165f218-f5c1-4171-9896-f46301b94e3a 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:11:55.059 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:11:55.060 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:55.060 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:55.060 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:55.060 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:11:55.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:55.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_max_scrubs 2024-01-10T07:11:55.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:11:55.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:11:55.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:11:55.272 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:11:55.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:11:55.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:11:55.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:11:55.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:11:55.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:11:55.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:55.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:11:55.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:11:55.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:55.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:11:55.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:11:55.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:11:55.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:11:55.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:55.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:55.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:11:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:11:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:11:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:11:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-01-10T07:11:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-01-10T07:11:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:11:55.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:11:55.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:11:55.287 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:11:55.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:11:55.287 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:11:55.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:11:55.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:11:55.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:11:55.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:55.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:11:55.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:11:55.291 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:55.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:11:55.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:11:55.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:11:55.293 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:11:55.293 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:55.293 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:55.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:11:55.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:11:55.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:11:55.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:11:55.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:11:55.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:55.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:55.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:11:55.297 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:11:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:11:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:11:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:11:55.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:11:55.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:11:55.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:55.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:55.651 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:55.651 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:55.652 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:55.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:55.652 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:11:55.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:11:55.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:11:55.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:55.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:55.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:11:55.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:11:55.905 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:55.905 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:55.905 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:55.905 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:55.906 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:55.906 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:55.906 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:55.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:55.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:11:56.011 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:11:56.011 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:56.011 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:56.011 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:56.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:11:56.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-01-10T07:11:56.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:11:56.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:11:56.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:11:56.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:11:56.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:11:56.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:11:57.023 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:11:57.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:57.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:57.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:57.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:57.201 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:57.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:57.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-01-10T07:11:57.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-01-10T07:11:57.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-01-10T07:11:57.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:11:57.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:11:57.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:11:57.441 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:11:57.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:11:57.442 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:11:57.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:11:57.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:11:57.443 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:11:57.445 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:57.445 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:11:57.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:11:57.446 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:57.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:11:57.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:11:57.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:11:57.449 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:11:57.449 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:57.450 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:57.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:11:57.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:11:57.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:11:57.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:11:57.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-01-10T07:11:57.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-01-10T07:11:57.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-01-10T07:11:57.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:11:57.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:11:57.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:11:57.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:11:57.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:11:57.456 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:11:57.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:11:57.456 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:11:57.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:11:57.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:11:57.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:11:57.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:11:57.463 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:11:57.463 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:57.463 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:57.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:11:57.464 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:11:57.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:11:57.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:11:57.467 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:11:57.467 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:11:57.467 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:11:57.467 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:11:57.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:11:57.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:11:57.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:11:57.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:11:57.827 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:11:57.827 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:57.828 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:57.828 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:57.828 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:57.828 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:57.828 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:57.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:11:58.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:11:58.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:11:58.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:58.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:58.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:11:58.097 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:11:58.097 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:58.097 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:58.097 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:58.097 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:58.098 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:58.098 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:58.098 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:58.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:58.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:11:58.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:11:58.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:11:58.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:11:58.201 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:11:58.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:11:58.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-01-10T07:11:58.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:11:58.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:11:58.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:11:58.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:11:58.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:11:58.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:11:59.199 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:11:59.199 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:11:59.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:11:59.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-01-10T07:11:59.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:11:59.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:11:59.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e5a7bc07-3f88-4c1a-a2f2-a7e536795072 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e5a7bc07-3f88-4c1a-a2f2-a7e536795072' 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 e5a7bc07-3f88-4c1a-a2f2-a7e536795072 2024-01-10T07:11:59.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:11:59.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/Q55lXcCPDhAAXGImRBDcMHkSdN/4wYJYIw== 2024-01-10T07:11:59.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/Q55lXcCPDhAAXGImRBDcMHkSdN/4wYJYIw=="}' 2024-01-10T07:11:59.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e5a7bc07-3f88-4c1a-a2f2-a7e536795072 -i td/ceph-helpers/0/new.json 2024-01-10T07:11:59.630 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:11:59.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:11:59.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/Q55lXcCPDhAAXGImRBDcMHkSdN/4wYJYIw== --osd-uuid e5a7bc07-3f88-4c1a-a2f2-a7e536795072 2024-01-10T07:11:59.668 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.667+0000 7f2be07a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:59.669 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.669+0000 7f2be07a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:59.670 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.670+0000 7f2be07a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:11:59.671 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.670+0000 7f2be07a6540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:11:59.671 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.670+0000 7f2be07a6540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:11:59.671 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.670+0000 7f2be07a6540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:11:59.721 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:11:59.721+0000 7f2be07a6540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:12:05.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:12:05.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:12:05.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:12:05.271 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:12:05.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:12:05.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:12:05.555 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:12:05.555 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:12:05.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:12:05.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:12:05.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:12:05.579 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:05.579+0000 7f366d8a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:05.593 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:05.593+0000 7f366d8a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:05.594 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:05.594+0000 7f366d8a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:12:06.068 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:12:06.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:06.069 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:06.137 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:06.137+0000 7f366d8a4540 -1 Falling back to public interface 2024-01-10T07:12:06.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:07.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:07.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:07.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:12:07.578 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:12:07.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:07.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:08.079 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:08.682 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:08.682+0000 7f366d8a4540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:12:09.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:09.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:09.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:12:09.081 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:12:09.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:09.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:09.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:10.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:10.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:10.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:12:10.689 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:12:10.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:10.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:11.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:12.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:12.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:12.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:12:12.151 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:12:12.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:12.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:12.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:13.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:13.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:13.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:12:13.716 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:12:13.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:13.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:14.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:15.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:15.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:15.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:12:15.189 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:12:15.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:15.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:15.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:16.700 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:16.700 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:16.700 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:12:16.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:12:16.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:16.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:17.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:18.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:18.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:18.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:12:18.161 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:12:18.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:18.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:18.800 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:19.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:19.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:19.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:12:19.802 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:12:19.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:19.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:20.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:21.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:21.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:21.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:12:21.743 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:12:21.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:21.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:22.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:23.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:23.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:23.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:12:23.722 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:12:23.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:23.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:24.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:25.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:25.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:25.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:12:25.373 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:12:25.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:25.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:25.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:26.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:26.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:26.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:12:26.956 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:12:26.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:26.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:27.685 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:28.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:28.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:28.687 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:12:28.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:12:28.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:28.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:29.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:30.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:30.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:30.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:12:30.643 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:12:30.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:30.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:31.137 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/103768,v1:127.0.0.1:6803/103768] [v2:127.0.0.1:6804/103768,v1:127.0.0.1:6805/103768] exists,up e5a7bc07-3f88-4c1a-a2f2-a7e536795072 2024-01-10T07:12:31.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:12:31.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:12:31.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:12:31.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-01-10T07:12:31.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:12:31.587 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:12:31.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:12:31.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:12:32.307 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:12:32.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:12:33.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:12:34.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-01-10T07:12:34.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:12:34.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:12:34.276 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:12:34.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:12:34.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:12:34.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:12:34.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:12:34.380 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:12:34.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:12:34.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:12:34.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:12:34.845 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836488 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836488 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836488' 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836488 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:12:34.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:12:34.984 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836488 2024-01-10T07:12:34.984 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:12:34.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836488 2024-01-10T07:12:34.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-01-10T07:12:34.985 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836488 2024-01-10T07:12:34.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:12:35.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836486 -lt 21474836488 2024-01-10T07:12:35.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:12:36.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:12:36.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:12:36.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836488 2024-01-10T07:12:36.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:12:36.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:12:36.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:12:37.622 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:12:38.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:12:38.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:12:38.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:12:38.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:12:38.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:12:38.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:12:38.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:12:38.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-01-10T07:12:38.718 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:12:38.719 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:12:38.719 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:12:38.719 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:12:39.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-01-10T07:12:31.958999+0000 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-01-10T07:12:31.958999+0000 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:12:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:12:39.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:12:39.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:12:39.264 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:12:39.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:12:39.264 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:12:39.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:12:39.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:12:39.266 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:12:39.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:12:39.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:12:39.268 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:12:39.268 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:12:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:12:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:12:39.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:12:39.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:12:39.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:39.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:39.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:12:39.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_not_primary td/ceph-helpers 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:12:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:12:39.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:12:39.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:12:39.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:12:39.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:12:39.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:12:39.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:12:39.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:12:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:12:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:12:39.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:12:39.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:12:39.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:12:39.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:12:39.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:12:39.285 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:12:39.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:39.286 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:39.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:12:39.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:12:39.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:12:39.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:12:39.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:12:39.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:39.288 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:39.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:12:39.290 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:12:39.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:12:39.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:12:39.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:12:39.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-01-10T07:12:39.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:12:39.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:12:39.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:12:39.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:12:39.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:12:39.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-01-10T07:12:39.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:12:39.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:12:39.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:12:39.654 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:12:39.654 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:39.654 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:39.654 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:12:39.654 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-01-10T07:12:39.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:12:39.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:12:39.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:12:39.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:12:39.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:12:39.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:12:39.916 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:12:39.916 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:12:39.916 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:12:39.917 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:12:39.917 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:39.917 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:39.917 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:12:39.917 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:12:39.917 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:12:40.024 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:12:40.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:12:40.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-01-10T07:12:40.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:12:40.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:12:40.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:12:40.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:12:40.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:12:40.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:12:41.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:12:41.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:12:41.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:12:41.025 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:12:41.026 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:41.026 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:41.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:12:41.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:12:41.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:12:41.054 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:12:41.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:12:41.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:12:41.057 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:12:41.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3709a13-202b-4b1b-8ca0-a6b8e9bc69fe 2024-01-10T07:12:41.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f3709a13-202b-4b1b-8ca0-a6b8e9bc69fe' 2024-01-10T07:12:41.058 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 f3709a13-202b-4b1b-8ca0-a6b8e9bc69fe 2024-01-10T07:12:41.058 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:12:41.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpQ55lmx5BBBAANiPAh+0yqmQsF/sggMAQSA== 2024-01-10T07:12:41.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpQ55lmx5BBBAANiPAh+0yqmQsF/sggMAQSA=="}' 2024-01-10T07:12:41.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3709a13-202b-4b1b-8ca0-a6b8e9bc69fe -i td/ceph-helpers/0/new.json 2024-01-10T07:12:41.457 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:12:41.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:12:41.469 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpQ55lmx5BBBAANiPAh+0yqmQsF/sggMAQSA== --osd-uuid f3709a13-202b-4b1b-8ca0-a6b8e9bc69fe 2024-01-10T07:12:41.494 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.493+0000 7f50a9a2b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:41.496 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.495+0000 7f50a9a2b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:41.497 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.496+0000 7f50a9a2b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:41.497 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.496+0000 7f50a9a2b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:12:41.498 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.496+0000 7f50a9a2b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:12:41.498 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.496+0000 7f50a9a2b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:12:41.576 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:41.574+0000 7f50a9a2b540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:12:46.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:12:46.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:12:46.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:12:46.830 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:12:46.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:12:47.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:12:47.074 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:12:47.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:12:47.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:12:47.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:12:47.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:12:47.099 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:47.098+0000 7fd54f9ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:47.110 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:47.110+0000 7fd54f9ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:47.111 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:47.111+0000 7fd54f9ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:12:47.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:12:47.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:12:47.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:12:47.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:12:47.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:12:47.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:47.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:12:47.353 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:12:47.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:47.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:47.640 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:47.640+0000 7fd54f9ac540 -1 Falling back to public interface 2024-01-10T07:12:47.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:48.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:48.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:48.863 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:12:48.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:12:48.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:48.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:49.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:50.076 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:12:50.076+0000 7fd54f9ac540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:12:50.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:50.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:50.332 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:12:50.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:12:50.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:50.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:50.800 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:51.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:51.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:51.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:12:51.802 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:12:51.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:51.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:52.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:53.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:53.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:53.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:12:53.262 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:12:53.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:53.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:53.723 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:54.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:54.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:54.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:12:54.725 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:12:54.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:54.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:55.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:56.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:56.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:56.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:12:56.240 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:12:56.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:56.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:56.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:57.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:57.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:57.715 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:12:57.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:12:57.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:57.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:58.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:12:59.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:12:59.253 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:12:59.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:12:59.254 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:12:59.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:12:59.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:12:59.716 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:00.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:00.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:00.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:13:00.718 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:13:00.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:00.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:01.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:02.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:02.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:02.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:13:02.303 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:13:02.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:02.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:02.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:03.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:03.843 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:03.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:13:03.844 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:13:03.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:03.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:04.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:05.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:05.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:05.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:13:05.692 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:13:05.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:05.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:06.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:07.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:07.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:07.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:13:07.387 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:13:07.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:07.388 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:08.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:09.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:09.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:09.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:13:09.132 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:13:09.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:09.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:09.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:10.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:10.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:10.872 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:13:10.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:13:10.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:10.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:11.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:12.435 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:12.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:12.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:13:12.436 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:13:12.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:12.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:13:12.923 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/106115,v1:127.0.0.1:6803/106115] [v2:127.0.0.1:6804/106115,v1:127.0.0.1:6805/106115] exists,up f3709a13-202b-4b1b-8ca0-a6b8e9bc69fe 2024-01-10T07:13:12.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:13:12.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:13:12.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1d37879-0f99-488a-a597-5f000402ce06 2024-01-10T07:13:12.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d1d37879-0f99-488a-a597-5f000402ce06' 2024-01-10T07:13:12.931 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 d1d37879-0f99-488a-a597-5f000402ce06 2024-01-10T07:13:12.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:13:12.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIQ55lto4vOBAAfrVJp5mrBiQ5LJARlZOA4w== 2024-01-10T07:13:12.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIQ55lto4vOBAAfrVJp5mrBiQ5LJARlZOA4w=="}' 2024-01-10T07:13:12.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1d37879-0f99-488a-a597-5f000402ce06 -i td/ceph-helpers/1/new.json 2024-01-10T07:13:13.563 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:13:13.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:13:13.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIQ55lto4vOBAAfrVJp5mrBiQ5LJARlZOA4w== --osd-uuid d1d37879-0f99-488a-a597-5f000402ce06 2024-01-10T07:13:13.601 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.601+0000 7f0ffc48a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:13:13.603 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.603+0000 7f0ffc48a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:13:13.604 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.603+0000 7f0ffc48a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:13:13.604 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.604+0000 7f0ffc48a540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:13:13.604 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.604+0000 7f0ffc48a540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:13:13.604 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.604+0000 7f0ffc48a540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:13:13.678 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:13.678+0000 7f0ffc48a540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:13:19.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:13:19.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:13:19.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:13:19.549 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:13:19.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:13:20.204 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:13:20.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:13:20.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:13:20.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:13:20.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:13:20.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:13:20.227 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:20.226+0000 7f12500a1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:13:20.229 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:20.229+0000 7f12500a1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:13:20.230 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:20.229+0000 7f12500a1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:20.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:13:20.707 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:13:20.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:20.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:20.752 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:20.752+0000 7f12500a1540 -1 Falling back to public interface 2024-01-10T07:13:21.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:22.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:22.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:22.176 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:13:22.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:13:22.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:22.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:22.676 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:23.222 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:13:23.221+0000 7f12500a1540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:13:23.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:23.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:23.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:13:23.678 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:13:23.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:23.679 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:24.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:25.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:25.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:25.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:13:25.291 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:13:25.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:25.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:25.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:26.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:26.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:26.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:13:26.753 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:13:26.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:26.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:27.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:28.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:28.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:28.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:13:28.303 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:13:28.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:28.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:28.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:29.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:29.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:29.793 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:13:29.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:13:29.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:29.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:30.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:31.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:31.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:31.385 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:13:31.386 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:13:31.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:31.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:31.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:32.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:32.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:32.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:13:32.853 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:13:32.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:32.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:33.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:34.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:34.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:34.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:13:34.319 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:13:34.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:34.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:35.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:36.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:36.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:36.122 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:13:36.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:13:36.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:36.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:36.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:37.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:37.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:37.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:13:37.801 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:13:37.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:37.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:38.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:39.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:39.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:39.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:13:39.545 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:13:39.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:39.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:40.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:41.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:41.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:41.168 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:13:41.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:13:41.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:41.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:42.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:43.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:43.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:43.017 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:13:43.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:13:43.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:43.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:43.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:44.700 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:44.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:13:44.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:13:44.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:44.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:45.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:13:46.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:13:46.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:13:46.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:13:46.323 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:13:46.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:13:46.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:13:46.816 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/107931,v1:127.0.0.1:6811/107931] [v2:127.0.0.1:6812/107931,v1:127.0.0.1:6813/107931] exists,up d1d37879-0f99-488a-a597-5f000402ce06 2024-01-10T07:13:46.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:13:46.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:13:46.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:13:46.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-01-10T07:13:46.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:13:47.283 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:13:47.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:13:47.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:13:47.972 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:13:47.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:13:48.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:13:50.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-01-10T07:13:50.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:13:50.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:13:50.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:13:50.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:13:50.515 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:13:50.516 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:13:50.516 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:13:50.516 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:13:50.516 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:13:50.516 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:13:50.620 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:13:50.620 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:13:50.620 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:13:50.620 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:13:50.621 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:13:51.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:13:51.101 INFO:tasks.workunit.client.0.smithi057.stderr:1' 2024-01-10T07:13:51.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:13:51.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:13:51.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:13:51.280 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836494 2024-01-10T07:13:51.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836494 2024-01-10T07:13:51.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836494' 2024-01-10T07:13:51.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:13:51.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:13:51.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672968 2024-01-10T07:13:51.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672968 2024-01-10T07:13:51.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836494 1-42949672968' 2024-01-10T07:13:51.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:13:51.444 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836494 2024-01-10T07:13:51.444 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:13:51.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:13:51.445 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836494 2024-01-10T07:13:51.445 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:13:51.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836494 2024-01-10T07:13:51.446 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836494 2024-01-10T07:13:51.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-01-10T07:13:51.446 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:13:51.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836494 2024-01-10T07:13:51.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:13:52.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:13:52.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:13:53.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836494 2024-01-10T07:13:53.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:13:54.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2024-01-10T07:13:54.379 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:13:54.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836495 -lt 21474836494 2024-01-10T07:13:54.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:13:54.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672968 2024-01-10T07:13:54.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:13:54.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:13:54.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672968 2024-01-10T07:13:54.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:13:54.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672968 2024-01-10T07:13:54.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-01-10T07:13:54.851 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672968 2024-01-10T07:13:54.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:13:55.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672969 -lt 42949672968 2024-01-10T07:13:55.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:13:55.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:13:55.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:13:56.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:13:56.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:13:56.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:13:56.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:13:56.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:13:56.083 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:13:56.084 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:13:56.084 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:13:56.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:13:56.530 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:13:56.530 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:13:56.530 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:13:57.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:13:57.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:13:57.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:13:57.152 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-01-10T07:13:57.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-01-10T07:13:57.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-01-10T07:13:57.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-01-10T07:13:57.153 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-01-10T07:13:57.604 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-01-10T07:13:58.106 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-01-10T07:13:58.106 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-01-10T07:13:58.106 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-01-10T07:13:58.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:13:58.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:13:58.724 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:13:58.724 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:13:58.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:13:58.725 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:13:58.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:13:58.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:13:58.726 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:13:58.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:13:58.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:13:58.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:13:58.728 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:13:58.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:13:58.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:13:58.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:13:58.738 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:13:58.738 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:58.739 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:58.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_num_active_clean td/ceph-helpers 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:13:58.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:13:58.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:13:58.742 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:13:58.742 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:13:58.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:13:58.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:13:58.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:13:58.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:13:58.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:13:58.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:13:58.743 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:13:58.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:13:58.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:13:58.744 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:13:58.746 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:13:58.746 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:13:58.746 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:13:58.746 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:13:58.747 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:13:58.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:13:58.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:13:58.749 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:13:58.749 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:58.749 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:58.749 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:13:58.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:13:58.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:13:58.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:13:58.751 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:13:58.751 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:58.751 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:58.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:13:58.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:13:58.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:13:58.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:13:58.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:13:58.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:13:59.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:13:59.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:13:59.303 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:13:59.303 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:13:59.303 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:13:59.303 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:13:59.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:13:59.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:13:59.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:13:59.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:13:59.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:13:59.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:13:59.415 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:13:59.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-01-10T07:13:59.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:13:59.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:13:59.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:13:59.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:13:59.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:13:59.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:14:00.421 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:14:00.421 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:14:00.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:14:00.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:00.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:14:00.453 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:14:00.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9db0f6c9-46cb-4f3a-b464-f7af2d2f874e 2024-01-10T07:14:00.454 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 9db0f6c9-46cb-4f3a-b464-f7af2d2f874e 2024-01-10T07:14:00.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9db0f6c9-46cb-4f3a-b464-f7af2d2f874e' 2024-01-10T07:14:00.454 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:14:00.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4Q55lO+rdGxAAqe4w9ONbxMwmpLhoY7z9sA== 2024-01-10T07:14:00.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4Q55lO+rdGxAAqe4w9ONbxMwmpLhoY7z9sA=="}' 2024-01-10T07:14:00.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9db0f6c9-46cb-4f3a-b464-f7af2d2f874e -i td/ceph-helpers/0/new.json 2024-01-10T07:14:00.877 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:14:00.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:14:00.890 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4Q55lO+rdGxAAqe4w9ONbxMwmpLhoY7z9sA== --osd-uuid 9db0f6c9-46cb-4f3a-b464-f7af2d2f874e 2024-01-10T07:14:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.916+0000 7f4b923f2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:00.918 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.918+0000 7f4b923f2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:00.919 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.919+0000 7f4b923f2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:00.920 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.919+0000 7f4b923f2540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:14:00.920 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.919+0000 7f4b923f2540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:14:00.920 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.919+0000 7f4b923f2540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:14:00.991 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:00.990+0000 7f4b923f2540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:14:06.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:14:06.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:14:06.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:14:06.257 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:14:06.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:14:06.535 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:14:06.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:14:06.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:14:06.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:14:06.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:14:06.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:14:06.559 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:06.558+0000 7fc9abeff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:06.597 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:06.596+0000 7fc9abeff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:06.597 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:06.597+0000 7fc9abeff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:06.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:14:06.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:14:06.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:14:06.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:14:06.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:14:06.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:06.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:14:06.823 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:14:06.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:06.824 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:07.137 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:07.137+0000 7fc9abeff540 -1 Falling back to public interface 2024-01-10T07:14:07.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:08.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:08.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:08.330 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:14:08.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:14:08.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:08.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:08.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:09.551 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:09.551+0000 7fc9abeff540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:14:09.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:09.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:09.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:14:09.807 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:14:09.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:09.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:10.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:11.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:11.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:11.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:14:11.398 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:14:11.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:11.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:11.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:12.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:12.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:12.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:14:12.868 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:14:12.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:12.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:13.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:14.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:14.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:14.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:14:14.362 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:14:14.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:14.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:14.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:15.838 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:15.838 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:15.838 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:14:15.838 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:14:15.838 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:15.838 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:16.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:17.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:17.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:17.357 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:14:17.358 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:14:17.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:17.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:17.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:18.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:18.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:18.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:14:18.828 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:14:18.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:18.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:19.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:20.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:20.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:20.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:14:20.351 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:14:20.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:20.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:20.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:21.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:21.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:21.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:14:21.886 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:14:21.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:21.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:22.982 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:23.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:23.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:23.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:14:23.984 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:14:23.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:23.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:24.800 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:25.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:25.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:25.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:14:25.802 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:14:25.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:25.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:26.711 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:27.712 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:27.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:27.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:14:27.713 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:14:27.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:27.713 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:28.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:29.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:29.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:29.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:14:29.239 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:14:29.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:29.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:29.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:30.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:30.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:30.762 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:14:30.762 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:14:30.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:30.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:31.359 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/110450,v1:127.0.0.1:6803/110450] [v2:127.0.0.1:6804/110450,v1:127.0.0.1:6805/110450] exists,up 9db0f6c9-46cb-4f3a-b464-f7af2d2f874e 2024-01-10T07:14:31.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:14:31.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:14:31.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:14:31.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-01-10T07:14:31.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:14:31.876 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:14:31.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:14:31.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:14:32.524 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:14:32.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:14:33.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:14:34.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-01-10T07:14:34.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:14:34.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:14:34.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:14:34.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:14:34.507 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:14:34.508 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:14:34.508 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:14:34.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:14:34.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:14:34.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:14:34.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:14:34.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:14:34.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:14:34.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:14:34.612 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:14:35.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:14:35.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:14:35.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:14:35.094 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:14:35.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836487 2024-01-10T07:14:35.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836487 2024-01-10T07:14:35.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836487' 2024-01-10T07:14:35.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:14:35.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836487 2024-01-10T07:14:35.273 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:14:35.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:14:35.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836487 2024-01-10T07:14:35.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:14:35.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836487 2024-01-10T07:14:35.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-01-10T07:14:35.275 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836487 2024-01-10T07:14:35.275 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:14:35.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836487 2024-01-10T07:14:35.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:14:36.779 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:14:36.780 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:14:37.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836487 2024-01-10T07:14:37.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:14:37.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:14:37.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:14:37.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:14:38.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:14:38.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:14:38.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:14:38.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:14:38.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:14:38.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:14:38.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:14:39.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-01-10T07:14:39.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-01-10T07:14:39.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-01-10T07:14:39.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:14:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:14:39.482 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:14:39.483 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:14:39.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:14:39.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:14:39.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:14:39.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:14:39.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:14:39.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:14:39.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:14:39.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:14:39.487 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:14:39.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:14:39.488 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:14:39.488 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:14:39.493 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:14:39.493 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:39.494 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:39.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:14:39.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:14:39.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:14:39.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:14:39.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:14:39.497 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:14:39.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:14:39.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:14:39.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:14:39.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:14:39.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:14:39.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:14:39.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:14:39.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:14:39.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:14:39.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:14:39.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:14:39.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:14:39.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:14:39.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:39.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:39.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:14:39.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:14:39.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:14:39.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:14:39.506 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:14:39.506 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:39.506 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:39.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:14:39.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:14:39.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:14:39.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:39.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:39.869 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:14:40.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:14:40.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:14:40.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:40.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:14:40.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:14:40.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:14:40.231 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:14:40.232 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:14:40.232 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:40.232 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:40.232 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:14:40.232 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:14:40.232 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:14:40.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:14:41.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:14:41.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:14:41.229 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:14:41.229 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:14:41.230 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:41.230 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:41.230 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:41.230 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:14:41.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:14:41.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-01-10T07:14:41.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:14:41.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:14:41.261 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:14:41.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:14:41.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:14:41.263 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:14:41.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=672bdd6b-2de0-469c-8ae6-c276c34a2db2 2024-01-10T07:14:41.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 672bdd6b-2de0-469c-8ae6-c276c34a2db2' 2024-01-10T07:14:41.263 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 672bdd6b-2de0-469c-8ae6-c276c34a2db2 2024-01-10T07:14:41.263 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:14:41.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhQ55lMYJbEBAAvEiEnVBXHCeJ1TEZy3LLAg== 2024-01-10T07:14:41.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhQ55lMYJbEBAAvEiEnVBXHCeJ1TEZy3LLAg=="}' 2024-01-10T07:14:41.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 672bdd6b-2de0-469c-8ae6-c276c34a2db2 -i td/ceph-helpers/0/new.json 2024-01-10T07:14:41.670 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:14:41.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:14:41.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhQ55lMYJbEBAAvEiEnVBXHCeJ1TEZy3LLAg== --osd-uuid 672bdd6b-2de0-469c-8ae6-c276c34a2db2 2024-01-10T07:14:41.708 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.707+0000 7f82887b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:41.710 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.709+0000 7f82887b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:41.710 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.710+0000 7f82887b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:41.711 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.711+0000 7f82887b5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:14:41.711 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.711+0000 7f82887b5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:14:41.711 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.711+0000 7f82887b5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:14:41.784 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:41.784+0000 7f82887b5540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:14:47.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:14:47.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:14:47.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:14:47.086 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:14:47.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:14:47.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:14:47.374 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:14:47.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:14:47.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:14:47.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:14:47.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:14:47.397 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:47.396+0000 7f3194280540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:47.472 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:47.471+0000 7f3194280540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:47.473 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:47.472+0000 7f3194280540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:47.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:14:47.657 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:14:47.657 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:47.657 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:48.001 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:48.000+0000 7f3194280540 -1 Falling back to public interface 2024-01-10T07:14:48.137 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:49.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:49.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:49.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:14:49.139 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:14:49.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:49.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:49.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:50.411 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:14:50.410+0000 7f3194280540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:14:50.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:50.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:50.625 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:14:50.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:14:50.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:50.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:51.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:52.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:52.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:52.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:14:52.127 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:14:52.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:52.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:52.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:53.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:53.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:53.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:14:53.587 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:14:53.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:53.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:54.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:55.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:55.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:55.100 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:14:55.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:14:55.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:55.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:55.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:56.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:56.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:56.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:14:56.615 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:14:56.615 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:56.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:57.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:58.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:58.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:58.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:14:58.130 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:14:58.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:58.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:14:58.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:14:59.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:14:59.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:14:59.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:14:59.592 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:14:59.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:14:59.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:00.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:01.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:01.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:01.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:15:01.100 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:15:01.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:01.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:01.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:02.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:02.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:02.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:15:02.754 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:15:02.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:02.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:03.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:04.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:04.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:04.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:15:04.849 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:15:04.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:04.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:05.726 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:06.727 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:06.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:06.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:15:06.728 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:15:06.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:06.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:07.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:08.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:08.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:08.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:15:08.664 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:15:08.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:08.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:09.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:10.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:10.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:10.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:15:10.639 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:15:10.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:10.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:11.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:12.157 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:12.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:12.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:15:12.158 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:15:12.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:12.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:12.645 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/112867,v1:127.0.0.1:6803/112867] [v2:127.0.0.1:6804/112867,v1:127.0.0.1:6805/112867] exists,up 672bdd6b-2de0-469c-8ae6-c276c34a2db2 2024-01-10T07:15:12.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:15:12.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:15:12.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:15:12.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-01-10T07:15:12.646 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:15:13.105 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:15:13.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:15:13.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:15:13.864 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:15:13.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:15:14.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:15:15.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-01-10T07:15:15.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:15:15.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:15:15.942 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:15:15.942 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:15:16.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:15:16.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:15:16.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:15:16.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:15:16.046 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:15:16.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:15:16.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:15:16.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:15:16.553 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:15:16.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836487 2024-01-10T07:15:16.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836487 2024-01-10T07:15:16.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836487' 2024-01-10T07:15:16.691 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:15:16.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836487 2024-01-10T07:15:16.691 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:15:16.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:15:16.692 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836487 2024-01-10T07:15:16.693 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:15:16.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836487 2024-01-10T07:15:16.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-01-10T07:15:16.694 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836487 2024-01-10T07:15:16.694 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:15:17.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836487 2024-01-10T07:15:17.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:15:18.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:15:18.148 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:15:18.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836487 2024-01-10T07:15:18.638 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:15:18.638 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:15:18.639 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:15:19.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:15:19.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:15:19.291 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:15:19.291 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:15:19.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:15:19.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:15:19.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:15:19.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:15:19.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:15:19.728 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:15:19.728 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:15:19.728 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:15:20.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:15:20.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:15:20.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:15:20.358 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-01-10T07:15:20.358 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-01-10T07:15:20.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-01-10T07:15:20.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-01-10T07:15:20.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-01-10T07:15:20.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-01-10T07:15:20.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-01-10T07:15:20.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:15:20.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:15:20.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:15:20.904 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:15:20.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:15:20.905 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:15:20.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:15:20.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:15:20.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:15:20.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:15:20.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:15:20.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:15:20.909 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:15:20.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:15:20.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:15:20.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:15:20.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:15:20.916 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:20.916 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:20.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_num_pgs td/ceph-helpers 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-01-10T07:15:20.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:15:20.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:15:20.920 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:15:20.920 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:15:20.920 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:15:20.920 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:15:20.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:15:20.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:15:20.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:15:20.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:15:20.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:15:20.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:15:20.923 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:15:20.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:15:20.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:15:20.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:15:20.926 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:15:20.926 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:20.926 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:20.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:15:20.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:15:20.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:15:20.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:15:20.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:15:20.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:20.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:20.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:15:20.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:15:20.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:15:21.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:15:21.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:15:21.547 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:15:21.547 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:15:21.548 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:15:21.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:15:21.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:15:21.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-01-10T07:15:21.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:15:21.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:15:21.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:15:21.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:15:21.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:15:21.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:15:22.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:15:22.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:15:22.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:15:22.656 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:15:22.656 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:22.657 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:22.657 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:15:22.657 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:15:22.657 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:15:22.686 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:15:22.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:15:22.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:15:22.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:15:22.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:15:22.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b902663d-3e03-4ff1-bae1-92c82e9186d4 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b902663d-3e03-4ff1-bae1-92c82e9186d4' 2024-01-10T07:15:22.689 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 b902663d-3e03-4ff1-bae1-92c82e9186d4 2024-01-10T07:15:22.690 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:15:22.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKRJ5l5ozUKRAA2TBC31GVdCf4uL7nii+e2g== 2024-01-10T07:15:22.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKRJ5l5ozUKRAA2TBC31GVdCf4uL7nii+e2g=="}' 2024-01-10T07:15:22.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b902663d-3e03-4ff1-bae1-92c82e9186d4 -i td/ceph-helpers/0/new.json 2024-01-10T07:15:23.105 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:15:23.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:15:23.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKRJ5l5ozUKRAA2TBC31GVdCf4uL7nii+e2g== --osd-uuid b902663d-3e03-4ff1-bae1-92c82e9186d4 2024-01-10T07:15:23.143 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.142+0000 7f85b620b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:15:23.144 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.144+0000 7f85b620b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:15:23.145 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.145+0000 7f85b620b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:15:23.146 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.145+0000 7f85b620b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:15:23.146 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.145+0000 7f85b620b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:15:23.146 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.145+0000 7f85b620b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:15:23.220 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:23.219+0000 7f85b620b540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:15:28.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:15:28.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:15:28.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:15:28.519 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:15:28.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:15:28.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:15:28.809 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:15:28.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:15:28.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:15:28.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:15:28.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:15:28.831 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:28.831+0000 7f47e6037540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:15:28.907 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:28.906+0000 7f47e6037540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:15:28.907 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:28.907+0000 7f47e6037540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:15:29.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:15:29.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:15:29.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:15:29.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:15:29.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:15:29.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:29.093 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:15:29.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:15:29.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:29.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:29.435 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:29.435+0000 7f47e6037540 -1 Falling back to public interface 2024-01-10T07:15:29.555 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:30.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:30.556 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:30.557 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:15:30.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:15:30.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:30.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:31.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:31.863 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:15:31.862+0000 7f47e6037540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:15:32.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:32.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:32.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:15:32.034 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:15:32.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:32.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:32.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:33.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:33.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:33.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:15:33.579 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:15:33.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:33.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:34.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:35.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:35.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:35.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:15:35.036 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:15:35.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:35.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:35.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:36.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:36.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:36.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:15:36.543 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:15:36.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:36.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:37.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:38.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:38.083 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:38.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:15:38.084 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:15:38.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:38.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:38.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:39.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:39.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:39.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:15:39.551 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:15:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:39.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:40.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:41.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:41.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:41.019 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:15:41.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:15:41.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:41.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:41.541 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:42.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:42.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:42.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:15:42.576 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:15:42.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:42.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:43.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:44.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:44.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:44.180 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:15:44.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:15:44.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:44.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:45.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:46.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:46.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:46.208 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:15:46.208 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:15:46.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:46.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:47.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:48.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:48.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:48.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:15:48.155 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:15:48.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:48.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:48.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:49.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:49.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:49.900 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:15:49.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:15:49.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:49.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:50.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:51.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:51.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:51.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:15:51.894 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:15:51.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:51.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:52.737 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:15:53.738 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:15:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:15:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:15:53.739 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:15:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:15:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:15:54.236 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/115285,v1:127.0.0.1:6803/115285] [v2:127.0.0.1:6804/115285,v1:127.0.0.1:6805/115285] exists,up b902663d-3e03-4ff1-bae1-92c82e9186d4 2024-01-10T07:15:54.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:15:54.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:15:54.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:15:54.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-01-10T07:15:54.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:15:54.711 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:15:54.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:15:54.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:15:55.539 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:15:55.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:15:56.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:15:57.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-01-10T07:15:57.616 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:15:57.617 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:15:57.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:15:57.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:15:57.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:15:57.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:15:57.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:15:58.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:15:58.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:15:58.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:15:58.195 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:15:58.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836485 2024-01-10T07:15:58.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836485 2024-01-10T07:15:58.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836485' 2024-01-10T07:15:58.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:15:58.334 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836485 2024-01-10T07:15:58.334 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:15:58.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:15:58.335 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836485 2024-01-10T07:15:58.335 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:15:58.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836485 2024-01-10T07:15:58.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-01-10T07:15:58.337 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836485 2024-01-10T07:15:58.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:15:58.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836483 -lt 21474836485 2024-01-10T07:15:58.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:15:59.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:15:59.797 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:16:00.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836485 2024-01-10T07:16:00.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:16:00.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:16:00.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:16:00.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:16:01.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:16:01.426 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:16:01.426 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:16:01.426 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:16:02.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:16:02.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:16:02.071 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:16:02.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-01-10T07:16:02.071 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:16:02.072 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:16:02.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:16:02.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:16:02.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:16:02.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:16:02.788 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:16:02.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:16:02.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:16:02.789 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:16:02.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:02.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:16:02.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:16:02.792 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:02.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:16:02.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:16:02.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:16:02.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:16:02.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:02.799 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:02.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_op_scheduler td/ceph-helpers 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: test_get_op_scheduler: local dir=td/ceph-helpers 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: test_get_op_scheduler: setup td/ceph-helpers 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:16:02.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:16:02.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:16:02.802 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:16:02.802 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:16:02.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:16:02.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:16:02.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:16:02.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:16:02.803 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:16:02.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:16:02.803 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:16:02.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:16:02.804 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:16:02.804 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:16:02.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:02.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:16:02.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:16:02.806 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:02.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:16:02.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:16:02.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:16:02.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:16:02.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:02.809 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:16:02.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:16:02.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:16:02.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:16:02.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:16:02.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:02.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:02.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2196: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:16:02.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:16:02.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:16:02.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:16:02.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:16:02.814 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:16:03.117 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:16:03.117 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:03.117 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:03.117 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:03.117 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:03.117 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:03.118 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:03.118 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:16:03.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:16:03.359 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:16:03.361 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:16:03.361 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:16:03.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:16:03.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:16:03.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:16:03.468 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:16:03.568 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-01-10T07:16:03.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:16:03.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:16:03.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:16:03.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:16:03.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:16:03.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:16:04.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:16:04.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:04.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:04.460 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:04.461 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:04.461 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:04.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:04.461 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:16:04.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:16:04.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-01-10T07:16:04.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:16:04.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:16:04.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:16:04.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:16:04.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:04.495 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:16:04.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:16:04.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-01-10T07:16:04.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:16:04.498 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:16:04.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de6e56d4-5e3b-43ee-92e3-e4f145218bd9 2024-01-10T07:16:04.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 de6e56d4-5e3b-43ee-92e3-e4f145218bd9' 2024-01-10T07:16:04.498 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 de6e56d4-5e3b-43ee-92e3-e4f145218bd9 2024-01-10T07:16:04.499 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:16:04.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0RJ5lilQcHhAAN5PuGElW6JErEkwk/RYFdg== 2024-01-10T07:16:04.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0RJ5lilQcHhAAN5PuGElW6JErEkwk/RYFdg=="}' 2024-01-10T07:16:04.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de6e56d4-5e3b-43ee-92e3-e4f145218bd9 -i td/ceph-helpers/0/new.json 2024-01-10T07:16:04.901 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:16:04.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:16:04.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQA0RJ5lilQcHhAAN5PuGElW6JErEkwk/RYFdg== --osd-uuid de6e56d4-5e3b-43ee-92e3-e4f145218bd9 2024-01-10T07:16:04.939 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:04.939+0000 7f750e6ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:04.941 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:04.941+0000 7f750e6ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:04.942 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:04.942+0000 7f750e6ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:04.942 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:04.942+0000 7f750e6ff540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:16:04.943 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:04.942+0000 7f750e6ff540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:16:04.943 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:04.942+0000 7f750e6ff540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:16:05.017 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:05.016+0000 7f750e6ff540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:16:10.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:16:10.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:16:10.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:16:10.572 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:16:10.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:16:10.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:16:10.875 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:16:10.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-01-10T07:16:10.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:16:10.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:16:10.876 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:16:10.901 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:10.901+0000 7f54935c4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:10.914 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:10.914+0000 7f54935c4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:10.915 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:10.915+0000 7f54935c4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:16:11.380 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:16:11.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:11.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:16:11.447 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:11.447+0000 7f54935c4540 -1 Falling back to public interface 2024-01-10T07:16:11.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:12.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:12.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:12.882 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:16:12.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:16:12.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:12.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:16:13.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:13.920 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:13.920+0000 7f54935c4540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:16:14.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:14.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:14.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:16:14.378 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:16:14.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:14.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:16:14.849 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/117742,v1:127.0.0.1:6803/117742] [v2:127.0.0.1:6804/117742,v1:127.0.0.1:6805/117742] exists,up de6e56d4-5e3b-43ee-92e3-e4f145218bd9 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_get_op_scheduler: get_op_scheduler 0 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: get_op_scheduler: local id=0 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: get_op_scheduler: get_config osd 0 osd_op_queue 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-01-10T07:16:14.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:16:14.852 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_op_queue 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_get_op_scheduler: test wpq = wpq 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:16:14.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:14.962 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=34ac81a9-7539-4b6d-b396-d54fef571708 2024-01-10T07:16:14.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 34ac81a9-7539-4b6d-b396-d54fef571708' 2024-01-10T07:16:14.964 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 34ac81a9-7539-4b6d-b396-d54fef571708 2024-01-10T07:16:14.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:16:14.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+RJ5lJYcyOhAAjCkxTcRLvuQlVtWX0ZkrrQ== 2024-01-10T07:16:14.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+RJ5lJYcyOhAAjCkxTcRLvuQlVtWX0ZkrrQ=="}' 2024-01-10T07:16:14.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 34ac81a9-7539-4b6d-b396-d54fef571708 -i td/ceph-helpers/1/new.json 2024-01-10T07:16:15.594 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:16:15.606 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:16:15.608 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQA+RJ5lJYcyOhAAjCkxTcRLvuQlVtWX0ZkrrQ== --osd-uuid 34ac81a9-7539-4b6d-b396-d54fef571708 2024-01-10T07:16:15.633 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.633+0000 7f889ca45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:15.639 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.638+0000 7f889ca45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:15.639 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.639+0000 7f889ca45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:15.640 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.640+0000 7f889ca45540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:16:15.640 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.640+0000 7f889ca45540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:16:15.640 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.640+0000 7f889ca45540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:16:15.731 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:15.731+0000 7f889ca45540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:16:21.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:16:21.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:16:21.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:16:21.305 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:16:21.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:16:21.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:16:21.933 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:16:21.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-01-10T07:16:21.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:16:21.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:16:21.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:16:21.956 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:21.956+0000 7fd6edb49540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:21.958 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:21.958+0000 7fd6edb49540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:21.959 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:21.958+0000 7fd6edb49540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:22.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:16:22.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:16:22.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:16:22.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:16:22.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:16:22.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:22.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:16:22.438 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:16:22.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:22.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:22.481 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:22.481+0000 7fd6edb49540 -1 Falling back to public interface 2024-01-10T07:16:22.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:23.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:23.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:23.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:16:23.915 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:16:23.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:23.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:24.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:25.005 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:25.005+0000 7fd6edb49540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:16:25.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:25.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:25.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:16:25.409 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:16:25.409 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:25.409 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:25.982 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:26.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:26.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:26.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:16:26.984 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:16:26.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:26.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:27.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:28.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:28.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:28.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:16:28.450 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:16:28.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:28.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:29.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:30.020 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:30.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:30.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:16:30.021 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:16:30.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:30.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:30.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:31.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:31.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:31.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:16:31.484 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:16:31.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:31.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:31.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:32.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:32.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:32.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:16:32.948 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:16:32.949 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:32.949 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:33.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:34.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:34.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:34.441 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:16:34.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:16:34.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:34.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:34.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:35.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:35.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:35.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:16:35.936 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:16:35.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:35.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:36.549 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:37.550 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:37.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:37.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:16:37.551 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:16:37.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:37.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:38.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:39.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:39.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:39.127 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:16:39.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:16:39.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:39.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:39.630 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:40.631 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:40.631 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:40.632 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:16:40.632 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:16:40.632 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:40.632 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:41.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:42.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:42.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:42.394 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:16:42.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:16:42.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:42.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:43.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:44.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:44.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:44.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:16:44.048 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:16:44.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:44.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:44.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:45.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:45.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:45.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:16:45.979 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:16:45.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:45.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:46.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:47.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:47.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:47.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:16:47.477 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:16:47.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:47.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:16:47.954 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/119128,v1:127.0.0.1:6811/119128] [v2:127.0.0.1:6812/119128,v1:127.0.0.1:6813/119128] exists,up 34ac81a9-7539-4b6d-b396-d54fef571708 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_get_op_scheduler: get_op_scheduler 1 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: get_op_scheduler: local id=1 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: get_op_scheduler: get_config osd 1 osd_op_queue 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-01-10T07:16:47.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.1.asok 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:16:47.957 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-osd.1.asok config get osd_op_queue 2024-01-10T07:16:48.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-01-10T07:16:48.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2204: test_get_op_scheduler: teardown td/ceph-helpers 2024-01-10T07:16:48.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:16:48.063 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:16:48.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:16:48.064 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:16:48.064 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:16:48.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:16:48.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:16:48.064 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:16:48.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:16:48.170 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:16:48.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:16:48.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:16:48.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:16:48.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:16:48.172 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:16:48.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:48.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:16:48.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:16:48.174 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:48.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:16:48.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:16:48.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:16:48.182 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:16:48.182 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:48.182 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:48.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:16:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:16:48.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:16:48.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:16:48.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:16:48.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:16:48.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:16:48.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:16:48.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:16:48.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:16:48.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:48.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:16:48.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:16:48.189 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:16:48.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:16:48.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:16:48.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:16:48.192 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:16:48.192 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:48.192 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:48.192 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:16:48.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:16:48.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:16:48.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:16:48.194 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:16:48.194 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:48.195 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:48.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:16:48.196 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:16:48.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:16:48.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:16:48.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:16:48.197 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:16:48.539 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:48.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:16:48.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:48.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:16:48.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:16:48.811 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:16:48.921 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:48.922 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:48.922 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:16:48.922 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:16:48.922 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:16:49.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-01-10T07:16:49.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:16:49.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:16:49.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:16:49.022 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:16:49.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:16:49.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:49.927 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:49.928 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:16:49.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:16:49.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:16:49.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:16:49.959 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:16:49.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84f67337-9c24-4d55-9b36-83797bed4b84 2024-01-10T07:16:49.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 84f67337-9c24-4d55-9b36-83797bed4b84' 2024-01-10T07:16:49.960 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 84f67337-9c24-4d55-9b36-83797bed4b84 2024-01-10T07:16:49.960 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:16:49.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhRJ5lBpYDOhAAahEcPOTV3ddTYFdo++Yqiw== 2024-01-10T07:16:49.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhRJ5lBpYDOhAAahEcPOTV3ddTYFdo++Yqiw=="}' 2024-01-10T07:16:49.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84f67337-9c24-4d55-9b36-83797bed4b84 -i td/ceph-helpers/0/new.json 2024-01-10T07:16:50.375 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:16:50.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:16:50.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhRJ5lBpYDOhAAahEcPOTV3ddTYFdo++Yqiw== --osd-uuid 84f67337-9c24-4d55-9b36-83797bed4b84 2024-01-10T07:16:50.412 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.412+0000 7f0d1fa03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:50.414 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.414+0000 7f0d1fa03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:50.415 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.415+0000 7f0d1fa03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:50.415 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.415+0000 7f0d1fa03540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:16:50.416 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.415+0000 7f0d1fa03540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:16:50.416 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.415+0000 7f0d1fa03540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:16:50.492 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:50.492+0000 7f0d1fa03540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:16:55.790 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:16:55.790 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:16:55.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:16:55.791 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:16:55.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:16:56.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:16:56.034 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:16:56.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:16:56.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:16:56.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:16:56.035 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:16:56.057 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:56.056+0000 7f1a45382540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:56.059 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:56.058+0000 7f1a45382540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:56.060 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:56.059+0000 7f1a45382540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:16:56.349 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:16:56.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:56.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:16:56.579 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:56.578+0000 7f1a45382540 -1 Falling back to public interface 2024-01-10T07:16:56.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:57.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:57.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:57.862 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:16:57.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:16:57.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:57.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:16:58.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:16:59.074 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:16:59.072+0000 7f1a45382540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:16:59.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:16:59.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:16:59.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:16:59.354 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:16:59.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:16:59.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:16:59.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:00.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:00.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:00.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:17:00.941 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:17:00.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:00.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:01.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:02.398 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:02.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:02.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:17:02.399 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:17:02.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:02.399 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:02.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:03.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:03.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:03.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:17:03.898 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:17:03.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:03.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:04.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:05.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:05.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:05.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:17:05.360 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:17:05.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:05.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:05.878 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:06.879 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:06.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:06.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:17:06.880 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:17:06.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:06.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:07.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:08.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:08.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:08.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:17:08.340 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:17:08.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:08.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:08.890 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:09.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:09.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:09.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:17:09.892 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:17:09.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:09.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:10.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:11.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:11.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:11.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:17:11.361 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:17:11.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:11.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:11.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:12.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:12.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:12.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:17:12.981 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:17:12.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:12.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:13.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:14.796 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:14.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:14.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:17:14.826 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:17:14.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:14.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:15.260 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:16.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:16.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:16.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:17:16.262 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:17:16.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:16.263 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:17.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:18.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:18.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:18.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:17:18.195 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:17:18.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:18.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:19.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:20.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:20.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:20.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:17:20.033 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:17:20.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:20.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:20.514 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/121143,v1:127.0.0.1:6803/121143] [v2:127.0.0.1:6804/121143,v1:127.0.0.1:6805/121143] exists,up 84f67337-9c24-4d55-9b36-83797bed4b84 2024-01-10T07:17:20.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:17:20.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:17:20.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:17:20.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-01-10T07:17:20.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:17:20.985 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:17:20.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:17:20.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:17:21.722 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:17:21.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:17:22.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:17:23.796 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-01-10T07:17:23.796 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:17:23.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:17:23.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:17:23.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:17:23.797 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:17:23.798 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:17:23.798 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:17:23.798 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:17:23.798 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:17:23.798 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:17:23.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:17:23.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:17:23.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:17:23.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:17:23.902 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:17:24.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:17:24.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:17:24.400 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:17:24.400 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:17:24.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836488 2024-01-10T07:17:24.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836488 2024-01-10T07:17:24.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836488' 2024-01-10T07:17:24.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:17:24.571 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836488 2024-01-10T07:17:24.571 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:17:24.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:17:24.572 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836488 2024-01-10T07:17:24.572 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:17:24.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836488 2024-01-10T07:17:24.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-01-10T07:17:24.574 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836488 2024-01-10T07:17:24.574 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:17:25.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836486 -lt 21474836488 2024-01-10T07:17:25.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:17:26.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:17:26.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:17:26.512 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836489 -lt 21474836488 2024-01-10T07:17:26.513 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:17:26.513 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:17:26.513 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:17:27.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:17:27.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:17:27.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:17:27.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:17:27.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:17:27.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:17:27.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:17:27.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:17:27.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:17:27.624 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:17:27.624 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:17:27.624 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:17:28.235 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:17:28.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:17:28.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:17:28.236 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-01-10T07:17:28.236 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-01-10T07:17:28.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-01-10T07:17:28.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-01-10T07:17:28.237 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-01-10T07:17:28.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-01-10T07:17:28.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-01-10T07:17:28.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-01-10T07:17:28.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:17:28.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:17:28.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:17:28.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:17:28.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:17:28.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:17:28.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:17:28.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:17:28.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:17:28.826 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:17:28.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:17:28.828 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:17:28.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:17:28.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:17:28.829 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:17:28.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:17:28.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:17:28.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:17:28.830 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:17:28.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:17:28.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:17:28.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:17:28.837 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:17:28.837 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:28.837 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:28.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_osds td/ceph-helpers 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-01-10T07:17:28.839 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:17:28.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:17:28.844 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:17:28.845 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:17:28.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:17:28.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:17:28.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:17:28.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:17:28.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:28.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:28.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:17:28.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:17:28.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:17:28.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:17:28.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:17:28.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:28.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:28.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:17:28.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:17:29.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-01-10T07:17:29.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:17:29.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:17:29.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:17:29.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:17:29.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:17:29.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:17:29.477 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:17:29.477 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:17:29.477 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:17:29.478 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:17:29.478 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:29.478 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:29.478 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:17:29.478 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:17:29.478 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:17:29.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:17:29.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:17:29.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:17:29.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:17:29.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-01-10T07:17:29.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:17:29.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:17:29.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:17:29.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:17:29.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:17:29.689 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:17:30.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:17:30.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:17:30.594 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:17:30.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:17:30.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:30.595 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:30.595 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:17:30.595 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:17:30.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:17:30.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-01-10T07:17:30.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:17:30.624 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:17:30.625 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=790cdf73-9a81-4a3a-a323-3aa0c3e0a748 2024-01-10T07:17:30.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 790cdf73-9a81-4a3a-a323-3aa0c3e0a748' 2024-01-10T07:17:30.627 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 790cdf73-9a81-4a3a-a323-3aa0c3e0a748 2024-01-10T07:17:30.627 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:17:30.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKRJ5lmdQnJhAAB5Z3DRMYzgPCFLuQanLgDA== 2024-01-10T07:17:30.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKRJ5lmdQnJhAAB5Z3DRMYzgPCFLuQanLgDA=="}' 2024-01-10T07:17:30.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 790cdf73-9a81-4a3a-a323-3aa0c3e0a748 -i td/ceph-helpers/0/new.json 2024-01-10T07:17:31.000 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:17:31.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:17:31.013 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKRJ5lmdQnJhAAB5Z3DRMYzgPCFLuQanLgDA== --osd-uuid 790cdf73-9a81-4a3a-a323-3aa0c3e0a748 2024-01-10T07:17:31.038 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.038+0000 7fc4397a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:17:31.040 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.040+0000 7fc4397a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:17:31.041 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.041+0000 7fc4397a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:17:31.042 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.041+0000 7fc4397a5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:17:31.042 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.041+0000 7fc4397a5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:17:31.042 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.041+0000 7fc4397a5540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:17:31.091 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:31.091+0000 7fc4397a5540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:17:36.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:17:36.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:17:36.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:17:36.407 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:17:36.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:17:36.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:17:36.696 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:17:36.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:17:36.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:17:36.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:17:36.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:17:36.720 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:36.719+0000 7fef364ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:17:36.762 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:36.761+0000 7fef364ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:17:36.763 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:36.763+0000 7fef364ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:17:36.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:17:36.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:17:36.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:17:36.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:17:36.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:17:36.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:36.970 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:17:36.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:17:36.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:36.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:37.293 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:37.293+0000 7fef364ac540 -1 Falling back to public interface 2024-01-10T07:17:37.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:38.435 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:38.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:38.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:17:38.436 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:17:38.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:38.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:38.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:39.708 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:17:39.708+0000 7fef364ac540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:17:39.920 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:39.920 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:39.921 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:17:39.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:17:39.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:39.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:40.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:41.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:41.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:41.415 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:17:41.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:17:41.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:41.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:41.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:42.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:42.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:42.882 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:17:42.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:17:42.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:42.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:43.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:44.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:44.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:44.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:17:44.351 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:17:44.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:44.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:44.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:45.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:45.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:45.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:17:45.842 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:17:45.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:45.842 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:46.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:47.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:47.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:47.353 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:17:47.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:17:47.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:47.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:47.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:48.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:48.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:48.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:17:48.825 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:17:48.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:48.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:49.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:50.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:50.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:50.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:17:50.367 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:17:50.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:50.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:50.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:51.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:51.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:51.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:17:51.904 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:17:51.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:51.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:52.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:53.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:53.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:53.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:17:53.542 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:17:53.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:53.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:54.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:55.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:55.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:55.302 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:17:55.302 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:17:55.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:55.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:55.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:56.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:56.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:56.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:17:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:17:56.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:56.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:57.674 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:17:58.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:17:58.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:17:58.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:17:58.676 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:17:58.676 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:17:58.676 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:17:59.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:00.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:00.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:00.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:18:00.352 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:18:00.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:00.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:18:00.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:01.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:01.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:01.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:18:01.848 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:18:01.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:01.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:18:02.342 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/123552,v1:127.0.0.1:6803/123552] [v2:127.0.0.1:6804/123552,v1:127.0.0.1:6805/123552] exists,up 790cdf73-9a81-4a3a-a323-3aa0c3e0a748 2024-01-10T07:18:02.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:18:02.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:18:02.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:18:02.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe000ca0-e674-4add-9ff1-66b87837d6b3 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fe000ca0-e674-4add-9ff1-66b87837d6b3' 2024-01-10T07:18:02.349 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 fe000ca0-e674-4add-9ff1-66b87837d6b3 2024-01-10T07:18:02.350 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:18:02.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqRJ5lqBuLFRAAvLR8hJH07uTVcCEJxUHXwQ== 2024-01-10T07:18:02.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqRJ5lqBuLFRAAvLR8hJH07uTVcCEJxUHXwQ=="}' 2024-01-10T07:18:02.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe000ca0-e674-4add-9ff1-66b87837d6b3 -i td/ceph-helpers/1/new.json 2024-01-10T07:18:03.030 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:18:03.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:18:03.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqRJ5lqBuLFRAAvLR8hJH07uTVcCEJxUHXwQ== --osd-uuid fe000ca0-e674-4add-9ff1-66b87837d6b3 2024-01-10T07:18:03.069 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.067+0000 7f223ae7f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:03.089 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.088+0000 7f223ae7f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:03.090 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.089+0000 7f223ae7f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:03.091 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.089+0000 7f223ae7f540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:18:03.091 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.089+0000 7f223ae7f540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:18:03.091 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.089+0000 7f223ae7f540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:18:03.182 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:03.181+0000 7f223ae7f540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:18:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:18:09.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:18:09.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:18:09.033 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:18:09.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:18:09.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:18:09.668 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:18:09.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:18:09.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:18:09.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:18:09.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:18:09.691 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:09.689+0000 7f84efd45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:09.724 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:09.723+0000 7f84efd45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:09.725 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:09.723+0000 7f84efd45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:10.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:18:10.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:18:10.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:18:10.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:18:10.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:18:10.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:10.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:18:10.139 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:18:10.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:10.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:10.245 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:10.244+0000 7f84efd45540 -1 Falling back to public interface 2024-01-10T07:18:10.597 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:11.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:11.599 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:11.599 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:18:11.599 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:18:11.600 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:11.600 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:12.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:12.688 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:12.687+0000 7f84efd45540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:18:13.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:13.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:13.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:18:13.084 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:18:13.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:13.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:13.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:14.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:14.558 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:14.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:18:14.559 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:18:14.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:14.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:15.080 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:16.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:16.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:16.081 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:18:16.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:18:16.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:16.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:16.631 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:17.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:17.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:17.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:18:17.633 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:18:17.633 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:17.634 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:18.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:19.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:19.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:19.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:18:19.095 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:18:19.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:19.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:19.643 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:20.644 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:20.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:20.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:18:20.645 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:18:20.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:20.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:21.110 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:22.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:22.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:22.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:18:22.114 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:18:22.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:22.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:22.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:23.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:23.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:23.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:18:23.613 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:18:23.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:23.614 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:24.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:25.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:25.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:25.130 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:18:25.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:18:25.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:25.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:25.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:26.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:26.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:26.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:18:26.944 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:18:26.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:26.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:27.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:28.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:28.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:28.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:18:28.639 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:18:28.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:28.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:29.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:30.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:30.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:30.219 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:18:30.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:18:30.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:30.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:31.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:32.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:32.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:32.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:18:32.152 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:18:32.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:32.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:33.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:34.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:34.141 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:18:34.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:18:34.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:34.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:34.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:35.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:35.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:35.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:18:35.794 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:18:35.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:35.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:18:36.260 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/125367,v1:127.0.0.1:6811/125367] [v2:127.0.0.1:6812/125367,v1:127.0.0.1:6813/125367] exists,up fe000ca0-e674-4add-9ff1-66b87837d6b3 2024-01-10T07:18:36.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:18:36.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:18:36.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:18:36.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-01-10T07:18:36.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:18:36.767 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:18:36.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:18:36.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:18:37.420 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:18:37.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:18:38.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:18:40.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-01-10T07:18:40.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:18:40.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:18:40.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:18:40.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:18:40.018 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:18:40.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:18:40.019 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:18:40.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:18:40.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:18:40.019 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:18:40.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:18:40.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:18:40.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:18:40.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:18:40.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:18:40.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:18:40.587 INFO:tasks.workunit.client.0.smithi057.stderr:1' 2024-01-10T07:18:40.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:18:40.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:18:40.588 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:18:40.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836492 2024-01-10T07:18:40.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836492 2024-01-10T07:18:40.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836492' 2024-01-10T07:18:40.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:18:40.725 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:18:40.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672968 2024-01-10T07:18:40.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672968 2024-01-10T07:18:40.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836492 1-42949672968' 2024-01-10T07:18:40.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:18:40.864 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836492 2024-01-10T07:18:40.864 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:18:40.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:18:40.865 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836492 2024-01-10T07:18:40.867 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:18:40.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836492 2024-01-10T07:18:40.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-01-10T07:18:40.868 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836492 2024-01-10T07:18:40.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:18:41.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836492 2024-01-10T07:18:41.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:18:41.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672968 2024-01-10T07:18:41.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:18:41.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:18:41.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672968 2024-01-10T07:18:41.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:18:41.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672968 2024-01-10T07:18:41.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-01-10T07:18:41.334 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 42949672968 2024-01-10T07:18:41.334 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:18:41.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672966 -lt 42949672968 2024-01-10T07:18:41.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:18:42.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:18:42.808 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:18:43.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672969 -lt 42949672968 2024-01-10T07:18:43.283 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:18:43.284 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:18:43.284 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:18:43.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:18:43.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:18:43.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:18:43.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:18:43.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:18:43.898 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:18:43.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:18:43.899 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:18:44.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:18:44.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:18:44.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:18:44.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:18:45.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:18:45.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:18:45.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:18:45.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-01-10T07:18:45.021 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:18:45.696 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:18:45.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:18:45.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:18:46.410 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:18:46.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:18:47.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:18:48.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-01-10T07:18:48.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-01-10T07:18:48.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-01-10T07:18:48.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-01-10T07:18:48.706 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-01-10T07:18:48.706 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-01-10T07:18:49.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-01-10T07:18:49.210 INFO:tasks.workunit.client.0.smithi057.stderr:0' 2024-01-10T07:18:49.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-01-10T07:18:49.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-01-10T07:18:49.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:18:49.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:18:49.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:18:49.211 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:18:49.211 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:18:49.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:18:49.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:18:49.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:18:49.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:18:49.316 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:18:49.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:18:49.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:18:49.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:18:49.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:18:49.319 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:18:49.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:18:49.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:18:49.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:18:49.321 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:18:49.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:18:49.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:18:49.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:18:49.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:18:49.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:49.332 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:49.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_pg td/ceph-helpers 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-01-10T07:18:49.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:18:49.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:18:49.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:18:49.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:18:49.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:18:49.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:18:49.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:18:49.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:18:49.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:18:49.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:18:49.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:18:49.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:18:49.339 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:18:49.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:18:49.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:18:49.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:18:49.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:18:49.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:49.342 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:49.342 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:18:49.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:18:49.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:18:49.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:18:49.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:18:49.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:49.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:49.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:18:49.347 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:18:49.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:18:49.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:18:49.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:18:49.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:18:49.655 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:18:49.655 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:18:49.655 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:18:49.656 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:18:49.656 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:49.656 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:49.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:18:49.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:18:49.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:18:49.906 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:18:49.907 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:18:49.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:18:49.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:49.908 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:49.908 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:18:49.908 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:18:49.908 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:18:50.011 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:18:50.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:18:50.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:18:50.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:18:50.012 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:18:50.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:18:50.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:18:50.013 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:18:50.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:18:50.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:50.013 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:50.014 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:18:50.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:18:50.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:18:50.111 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-01-10T07:18:50.111 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:18:50.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:18:50.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:18:50.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:18:50.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:18:50.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:18:51.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:18:51.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:18:51.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:18:51.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-01-10T07:18:51.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:18:51.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:18:51.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:18:51.036 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:18:51.037 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:18:51.038 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:18:51.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:18:51.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:18:51.039 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:18:51.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=216cd53d-c50d-45dc-9af1-e188151bc4f6 2024-01-10T07:18:51.039 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 216cd53d-c50d-45dc-9af1-e188151bc4f6' 2024-01-10T07:18:51.040 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 216cd53d-c50d-45dc-9af1-e188151bc4f6 2024-01-10T07:18:51.040 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:18:51.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbRJ5ldsAnAxAABokyk8o2OSs0Yh0/jE9v1A== 2024-01-10T07:18:51.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbRJ5ldsAnAxAABokyk8o2OSs0Yh0/jE9v1A=="}' 2024-01-10T07:18:51.053 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 216cd53d-c50d-45dc-9af1-e188151bc4f6 -i td/ceph-helpers/0/new.json 2024-01-10T07:18:51.463 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:18:51.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:18:51.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbRJ5ldsAnAxAABokyk8o2OSs0Yh0/jE9v1A== --osd-uuid 216cd53d-c50d-45dc-9af1-e188151bc4f6 2024-01-10T07:18:51.500 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.500+0000 7f3cdeff7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:51.502 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.502+0000 7f3cdeff7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:51.503 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.503+0000 7f3cdeff7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:51.504 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.503+0000 7f3cdeff7540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:18:51.504 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.503+0000 7f3cdeff7540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:18:51.504 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.503+0000 7f3cdeff7540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:18:51.577 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:51.574+0000 7f3cdeff7540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:18:56.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:18:56.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:18:56.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:18:56.868 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:18:56.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:18:57.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:18:57.161 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:18:57.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:18:57.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:18:57.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:18:57.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:18:57.184 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:57.183+0000 7f3d4465f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:57.242+0000 7f3d4465f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:57.243+0000 7f3d4465f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:18:57.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:18:57.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:18:57.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:18:57.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:18:57.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:18:57.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:57.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:18:57.459 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:18:57.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:57.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:18:57.772 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:18:57.772+0000 7f3d4465f540 -1 Falling back to public interface 2024-01-10T07:18:57.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:18:58.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:18:58.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:18:58.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:18:58.939 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:18:58.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:18:58.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:18:59.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:00.199 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:00.199+0000 7f3d4465f540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:19:00.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:00.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:00.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:19:00.421 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:19:00.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:00.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:00.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:01.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:01.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:01.910 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:19:01.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:19:01.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:01.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:02.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:03.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:03.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:03.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:19:03.370 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:19:03.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:03.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:03.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:04.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:04.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:04.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:19:04.868 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:19:04.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:04.869 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:05.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:06.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:06.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:06.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:19:06.407 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:19:06.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:06.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:06.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:07.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:07.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:07.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:19:07.888 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:19:07.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:07.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:08.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:09.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:09.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:09.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:19:09.358 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:19:09.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:09.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:09.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:10.882 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:10.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:10.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:19:10.883 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:19:10.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:10.883 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:11.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:12.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:12.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:12.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:19:12.547 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:19:12.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:12.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:13.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:14.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:14.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:14.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:19:14.591 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:19:14.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:14.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:15.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:16.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:16.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:16.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:19:16.474 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:19:16.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:16.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:17.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:18.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:18.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:18.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:19:18.180 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:19:18.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:18.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:18.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:19.873 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:19.873 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:19.873 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:19:19.874 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:19:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:19.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:20.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:21.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:21.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:21.510 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:19:21.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:19:21.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:21.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:21.993 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127878,v1:127.0.0.1:6803/127878] [v2:127.0.0.1:6804/127878,v1:127.0.0.1:6805/127878] exists,up 216cd53d-c50d-45dc-9af1-e188151bc4f6 2024-01-10T07:19:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:19:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:19:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:19:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-01-10T07:19:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:19:22.487 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:19:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:19:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:19:23.256 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:19:23.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:19:24.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:19:25.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-01-10T07:19:25.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:19:25.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:19:25.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:19:25.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:19:25.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:19:25.220 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:19:25.220 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:19:25.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:19:25.221 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:19:25.221 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:19:25.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:19:25.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:19:25.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:19:25.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:19:25.325 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:19:25.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:19:25.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:19:25.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:19:25.801 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:19:25.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836483 2024-01-10T07:19:25.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836483 2024-01-10T07:19:25.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836483' 2024-01-10T07:19:25.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:19:25.940 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836483 2024-01-10T07:19:25.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:19:25.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:19:25.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836483 2024-01-10T07:19:25.941 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:19:25.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836483 2024-01-10T07:19:25.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-01-10T07:19:25.943 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836483 2024-01-10T07:19:25.943 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:19:26.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836481 -lt 21474836483 2024-01-10T07:19:26.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:19:27.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:19:27.412 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:19:27.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836483 -lt 21474836483 2024-01-10T07:19:27.895 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:19:27.896 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:19:27.896 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:19:28.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:19:28.521 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:19:28.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:19:28.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:19:28.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:19:28.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-01-10T07:19:29.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-01-10T07:19:30.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-01-10T07:19:30.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:19:30.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:19:30.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:19:30.094 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:19:30.094 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:19:30.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:19:30.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:19:30.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:19:30.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:19:30.198 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:19:30.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:19:30.199 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:19:30.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:19:30.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:19:30.200 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:19:30.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:19:30.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:19:30.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:19:30.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:19:30.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:19:30.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:19:30.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:19:30.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:19:30.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:30.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:30.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_primary td/ceph-helpers 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:19:30.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:19:30.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:19:30.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:19:30.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:19:30.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:19:30.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:19:30.215 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:19:30.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:19:30.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:19:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:19:30.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:19:30.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:19:30.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:19:30.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:19:30.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:19:30.220 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:30.221 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:30.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:19:30.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:19:30.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:19:30.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:19:30.223 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:19:30.223 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:30.223 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:30.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:19:30.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:19:30.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:19:30.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:19:30.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:19:30.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:19:30.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:19:30.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:19:30.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:19:30.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:19:30.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:19:30.850 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:19:30.850 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:19:30.851 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:19:30.951 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:19:30.952 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:19:30.952 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:19:30.953 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:19:31.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-01-10T07:19:31.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-01-10T07:19:31.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:19:31.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:19:31.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:19:31.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:19:31.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:19:31.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:31.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:19:31.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:19:31.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:19:31.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-01-10T07:19:31.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:19:31.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:19:31.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:19:31.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:19:31.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:19:31.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:19:31.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:19:31.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:19:31.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:19:31.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49761fd2-dfdb-45ba-a9fd-3e501b922cd3 2024-01-10T07:19:31.987 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 49761fd2-dfdb-45ba-a9fd-3e501b922cd3 2024-01-10T07:19:31.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 49761fd2-dfdb-45ba-a9fd-3e501b922cd3' 2024-01-10T07:19:31.988 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:19:32.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAERZ5lRkEQABAAiqYdfjy6eBxA9Bbi2OjHEQ== 2024-01-10T07:19:32.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAERZ5lRkEQABAAiqYdfjy6eBxA9Bbi2OjHEQ=="}' 2024-01-10T07:19:32.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49761fd2-dfdb-45ba-a9fd-3e501b922cd3 -i td/ceph-helpers/0/new.json 2024-01-10T07:19:32.453 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:19:32.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:19:32.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAERZ5lRkEQABAAiqYdfjy6eBxA9Bbi2OjHEQ== --osd-uuid 49761fd2-dfdb-45ba-a9fd-3e501b922cd3 2024-01-10T07:19:32.491 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.490+0000 7f66f3f95540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:19:32.492 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.492+0000 7f66f3f95540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:19:32.493 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.493+0000 7f66f3f95540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:19:32.493 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.493+0000 7f66f3f95540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:19:32.494 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.493+0000 7f66f3f95540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:19:32.494 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.493+0000 7f66f3f95540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:19:32.568 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:32.568+0000 7f66f3f95540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:19:37.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:19:37.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:19:37.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:19:37.918 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:19:37.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:19:38.198 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:19:38.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:19:38.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:19:38.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:19:38.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:19:38.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:19:38.222 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:38.222+0000 7f7b0f2dd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:19:38.280 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:38.280+0000 7f7b0f2dd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:19:38.281 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:38.281+0000 7f7b0f2dd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:19:38.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:19:38.498 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:19:38.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:38.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:38.811 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:38.811+0000 7f7b0f2dd540 -1 Falling back to public interface 2024-01-10T07:19:38.982 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:39.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:39.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:39.983 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:19:39.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:19:39.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:39.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:40.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:41.236 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:19:41.235+0000 7f7b0f2dd540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:19:41.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:41.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:41.458 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:19:41.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:19:41.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:41.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:41.960 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:42.961 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:42.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:42.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:19:42.962 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:19:42.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:42.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:43.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:44.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:44.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:44.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:19:44.421 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:19:44.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:44.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:44.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:45.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:45.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:45.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:19:45.891 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:19:45.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:45.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:46.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:47.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:47.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:47.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:19:47.355 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:19:47.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:47.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:47.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:48.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:48.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:48.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:19:48.914 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:19:48.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:48.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:49.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:50.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:50.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:50.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:19:50.379 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:19:50.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:50.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:50.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:51.904 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:51.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:51.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:19:51.905 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:19:51.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:51.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:52.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:53.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:53.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:53.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:19:53.368 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:19:53.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:53.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:54.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:55.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:55.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:55.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:19:55.123 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:19:55.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:55.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:55.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:56.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:56.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:56.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:19:56.768 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:19:56.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:56.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:57.526 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:19:58.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:19:58.527 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:19:58.527 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:19:58.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:19:58.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:19:58.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:19:59.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:00.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:00.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:00.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:20:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:20:00.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:00.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:01.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:02.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:02.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:02.316 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:20:02.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:20:02.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:02.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:02.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:03.851 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:03.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:03.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:20:03.852 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:20:03.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:03.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:04.321 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/130286,v1:127.0.0.1:6803/130286] [v2:127.0.0.1:6804/130286,v1:127.0.0.1:6805/130286] exists,up 49761fd2-dfdb-45ba-a9fd-3e501b922cd3 2024-01-10T07:20:04.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:20:04.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:20:04.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:20:04.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-01-10T07:20:04.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:20:04.795 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:20:04.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:20:04.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:20:05.741 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:20:05.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:20:06.752 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:20:07.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-01-10T07:20:07.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:20:07.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:20:07.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:20:07.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:20:07.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:20:07.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:07.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:07.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:07.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:07.697 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:07.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:20:07.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:20:07.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:20:07.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:20:07.802 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:20:08.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:20:08.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:20:08.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:20:08.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:20:08.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836487 2024-01-10T07:20:08.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836487 2024-01-10T07:20:08.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836487' 2024-01-10T07:20:08.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:20:08.407 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836487 2024-01-10T07:20:08.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:20:08.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:20:08.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836487 2024-01-10T07:20:08.409 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:20:08.410 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836487 2024-01-10T07:20:08.410 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-01-10T07:20:08.410 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836487 2024-01-10T07:20:08.410 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:20:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836487 2024-01-10T07:20:08.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:20:09.872 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:20:09.872 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:20:10.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836487 2024-01-10T07:20:10.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:20:11.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2024-01-10T07:20:11.349 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:20:11.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836487 2024-01-10T07:20:11.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:20:11.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:20:11.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:20:12.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:20:12.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:20:12.493 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:20:12.493 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:20:12.493 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:20:12.493 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:20:12.494 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:20:12.494 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:20:12.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:20:12.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:20:12.935 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:20:12.935 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:20:13.551 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-01-10T07:20:13.552 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:20:14.055 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:20:14.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:20:14.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:20:14.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:20:14.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:20:14.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:20:14.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:20:14.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:20:14.163 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:20:14.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:14.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:20:14.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:20:14.165 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:14.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:20:14.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:20:14.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:20:14.172 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:20:14.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:14.173 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:14.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:20:14.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:20:14.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:20:14.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:20:14.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_timeout_delays td/ceph-helpers 2024-01-10T07:20:14.176 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-01-10T07:20:14.176 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.176 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.176 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.176 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.176 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-01-10T07:20:14.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-01-10T07:20:14.186 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.186 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-01-10T07:20:14.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-01-10T07:20:14.206 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.206 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-01-10T07:20:14.226 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-01-10T07:20:14.227 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.227 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.227 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.227 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.227 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-01-10T07:20:14.252 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-01-10T07:20:14.252 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.252 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.253 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.253 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.253 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-01-10T07:20:14.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-01-10T07:20:14.280 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.280 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.280 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-01-10T07:20:14.308 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-01-10T07:20:14.308 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.308 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.308 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.308 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.308 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-01-10T07:20:14.340 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-01-10T07:20:14.341 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.341 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.341 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.341 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.341 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-01-10T07:20:14.382 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-01-10T07:20:14.382 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.383 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.383 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.383 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.383 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-01-10T07:20:14.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-01-10T07:20:14.424 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.424 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.424 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.424 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.425 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.466 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-01-10T07:20:14.466 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-01-10T07:20:14.466 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.466 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.466 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.466 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-01-10T07:20:14.522 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-01-10T07:20:14.522 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.522 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.522 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.522 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.522 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-01-10T07:20:14.604 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-01-10T07:20:14.605 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:14.605 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:14.605 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:14.605 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:14.605 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:14.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-01-10T07:20:14.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:20:14.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_is_clean td/ceph-helpers 2024-01-10T07:20:14.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-01-10T07:20:14.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:20:14.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:20:14.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:20:14.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:20:14.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:20:14.861 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:20:14.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:20:14.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:20:14.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:20:14.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:14.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:20:14.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:20:14.866 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:14.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:20:14.867 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:20:14.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:20:14.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:20:14.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:14.868 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:14.868 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:20:14.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:20:14.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:20:14.870 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:20:14.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:20:14.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:14.871 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:14.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:20:14.873 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:20:14.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:20:14.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:20:14.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:20:14.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:20:15.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:20:15.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:20:15.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:20:15.166 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:20:15.167 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:15.167 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:15.167 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:20:15.167 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:20:15.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:20:15.407 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:20:15.408 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:20:15.409 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:20:15.409 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:15.409 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:15.409 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:20:15.409 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:20:15.409 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:20:15.510 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:20:15.510 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:20:15.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:20:15.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:20:16.500 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:20:16.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:20:16.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:20:16.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:20:16.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:20:16.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:20:16.533 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=162c5e07-498f-496e-8b2e-64851bba2d93 2024-01-10T07:20:16.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 162c5e07-498f-496e-8b2e-64851bba2d93' 2024-01-10T07:20:16.534 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 162c5e07-498f-496e-8b2e-64851bba2d93 2024-01-10T07:20:16.534 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:20:16.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwRZ5lnmScIBAANpgfBRQE5gLiOLzVxQntmQ== 2024-01-10T07:20:16.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwRZ5lnmScIBAANpgfBRQE5gLiOLzVxQntmQ=="}' 2024-01-10T07:20:16.548 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 162c5e07-498f-496e-8b2e-64851bba2d93 -i td/ceph-helpers/0/new.json 2024-01-10T07:20:16.941 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:20:16.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:20:16.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwRZ5lnmScIBAANpgfBRQE5gLiOLzVxQntmQ== --osd-uuid 162c5e07-498f-496e-8b2e-64851bba2d93 2024-01-10T07:20:16.978 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:16.977+0000 7ff0b8591540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:20:16.980 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:16.979+0000 7ff0b8591540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:20:16.981 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:16.980+0000 7ff0b8591540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:20:16.981 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:16.980+0000 7ff0b8591540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:20:16.981 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:16.980+0000 7ff0b8591540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:20:16.981 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:16.980+0000 7ff0b8591540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:20:17.053 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:17.052+0000 7ff0b8591540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:20:22.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:20:22.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:20:22.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:20:22.335 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:20:22.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:20:22.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:20:22.580 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:20:22.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:20:22.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:20:22.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:20:22.581 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:20:22.604 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:22.603+0000 7f48d469f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:20:22.607 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:22.605+0000 7f48d469f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:20:22.608 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:22.606+0000 7f48d469f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:20:22.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:20:22.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:20:22.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:20:22.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:20:22.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:20:22.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:22.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:20:22.848 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:20:22.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:22.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:23.125 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:23.124+0000 7f48d469f540 -1 Falling back to public interface 2024-01-10T07:20:23.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:24.359 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:20:24.360 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:20:24.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:24.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:24.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:25.556 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:20:25.556+0000 7f48d469f540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:20:25.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:25.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:25.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:20:25.837 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:20:25.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:25.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:26.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:27.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:27.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:27.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:20:27.374 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:20:27.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:27.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:27.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:28.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:28.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:28.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:20:28.836 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:20:28.836 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:28.837 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:29.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:30.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:30.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:30.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:20:30.340 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:20:30.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:30.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:30.803 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:31.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:31.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:31.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:20:31.805 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:20:31.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:31.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:32.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:33.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:33.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:33.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:20:33.290 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:20:33.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:33.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:33.753 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:34.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:34.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:34.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:20:34.755 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:20:34.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:34.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:35.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:36.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:36.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:36.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:20:36.307 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:20:36.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:36.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:36.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:37.768 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:37.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:37.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:20:37.769 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:20:37.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:37.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:38.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:39.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:39.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:39.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:20:39.472 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:20:39.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:39.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:40.108 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:41.109 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:41.109 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:41.110 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:20:41.110 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:20:41.110 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:41.110 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:41.619 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:42.621 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:42.621 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:42.621 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:20:42.621 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:20:42.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:42.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:43.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:44.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:44.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:44.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:20:44.413 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:20:44.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:44.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:45.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:46.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:46.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:46.397 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:20:46.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:20:46.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:46.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:47.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:20:48.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:20:48.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:20:48.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:20:48.183 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:20:48.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:20:48.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:20:48.663 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/133540,v1:127.0.0.1:6803/133540] [v2:127.0.0.1:6804/133540,v1:127.0.0.1:6805/133540] exists,up 162c5e07-498f-496e-8b2e-64851bba2d93 2024-01-10T07:20:48.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:20:48.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:20:48.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:20:48.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-01-10T07:20:48.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:20:49.180 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:20:49.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:20:49.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:20:50.090 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:20:50.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:20:51.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:20:52.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-01-10T07:20:52.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:20:52.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:20:52.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:20:52.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:20:52.074 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:20:52.075 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:20:52.075 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:20:52.075 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:20:52.075 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:20:52.075 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:20:52.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:20:52.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:20:52.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:20:52.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:20:52.179 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:20:52.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:20:52.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:20:52.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:20:52.669 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:20:52.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836485 2024-01-10T07:20:52.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836485 2024-01-10T07:20:52.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836485' 2024-01-10T07:20:52.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:20:52.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836485 2024-01-10T07:20:52.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:20:52.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:20:52.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836485 2024-01-10T07:20:52.848 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:20:52.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836485 2024-01-10T07:20:52.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-01-10T07:20:52.849 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836485 2024-01-10T07:20:52.849 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:20:53.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836483 -lt 21474836485 2024-01-10T07:20:53.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:20:54.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:20:54.313 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:20:54.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836483 -lt 21474836485 2024-01-10T07:20:54.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:20:55.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2024-01-10T07:20:55.789 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:20:56.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836486 -lt 21474836485 2024-01-10T07:20:56.266 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:20:56.267 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:20:56.267 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:20:56.925 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:20:57.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:20:57.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:20:57.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:20:57.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:20:57.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:20:57.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:20:57.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:20:57.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-01-10T07:20:57.982 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-01-10T07:20:57.982 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:20:57.982 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:20:58.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:20:59.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:20:59.026 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:20:59.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:20:59.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:20:59.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:20:59.132 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:20:59.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:20:59.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:20:59.134 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:20:59.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:59.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:20:59.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:20:59.136 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:59.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:20:59.137 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:20:59.137 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:20:59.143 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:20:59.143 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:59.143 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:59.143 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_kill_daemon td/ceph-helpers 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:20:59.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:20:59.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:20:59.147 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:20:59.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:20:59.147 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:20:59.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:20:59.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:20:59.149 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:20:59.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:59.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:20:59.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:20:59.151 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:20:59.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:20:59.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:20:59.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:20:59.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:20:59.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:59.154 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:59.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:20:59.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:20:59.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:20:59.155 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:20:59.156 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:20:59.156 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:59.156 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:59.156 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:20:59.158 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:20:59.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:20:59.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:20:59.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:20:59.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:20:59.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:20:59.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:20:59.785 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:20:59.785 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:59.785 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:59.785 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:20:59.785 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:20:59.786 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:20:59.891 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:20:59.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:20:59.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-01-10T07:20:59.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:20:59.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:20:59.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:20:59.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:20:59.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:20:59.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:00.885 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:21:00.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:21:00.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-01-10T07:21:00.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:21:00.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:21:00.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:21:00.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:21:00.915 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:21:00.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:21:00.917 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:21:00.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce1d7a84-cfa4-42cd-ad32-b529b21c1077 2024-01-10T07:21:00.917 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce1d7a84-cfa4-42cd-ad32-b529b21c1077' 2024-01-10T07:21:00.917 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 ce1d7a84-cfa4-42cd-ad32-b529b21c1077 2024-01-10T07:21:00.918 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:21:00.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcRZ5l/Wp4NxAA9ytLv1ZFzFMrQcTcMyynQg== 2024-01-10T07:21:00.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcRZ5l/Wp4NxAA9ytLv1ZFzFMrQcTcMyynQg=="}' 2024-01-10T07:21:00.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce1d7a84-cfa4-42cd-ad32-b529b21c1077 -i td/ceph-helpers/0/new.json 2024-01-10T07:21:01.333 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:21:01.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:21:01.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcRZ5l/Wp4NxAA9ytLv1ZFzFMrQcTcMyynQg== --osd-uuid ce1d7a84-cfa4-42cd-ad32-b529b21c1077 2024-01-10T07:21:01.370 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.370+0000 7f53325d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:01.372 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.372+0000 7f53325d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:01.373 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.373+0000 7f53325d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:01.373 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.373+0000 7f53325d3540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:21:01.374 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.373+0000 7f53325d3540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:21:01.374 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.373+0000 7f53325d3540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:21:01.446 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:01.446+0000 7f53325d3540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:21:06.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:21:06.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:21:06.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:21:06.734 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:21:06.734 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:21:07.011 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:21:07.012 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:21:07.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:21:07.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:21:07.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:21:07.012 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:21:07.035 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:07.035+0000 7f05404c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:07.176 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:07.176+0000 7f05404c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:07.177 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:07.177+0000 7f05404c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:07.350 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:21:07.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:21:07.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:07.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:07.705+0000 7f05404c6540 -1 Falling back to public interface 2024-01-10T07:21:07.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:08.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:08.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:08.834 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:21:08.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:21:08.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:08.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:09.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:10.116 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:10.115+0000 7f05404c6540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:21:10.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:10.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:10.314 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:21:10.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:21:10.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:10.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:10.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:11.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:11.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:11.833 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:21:11.833 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:21:11.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:11.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:12.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:13.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:13.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:13.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:21:13.296 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:21:13.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:13.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:13.769 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:14.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:14.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:14.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:21:14.771 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:21:14.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:14.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:15.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:16.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:16.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:16.301 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:21:16.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:21:16.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:16.301 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:16.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:17.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:17.766 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:17.766 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:21:17.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:21:17.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:17.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:18.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:19.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:19.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:19.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:21:19.231 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:21:19.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:19.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:19.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:20.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:20.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:20.733 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:21:20.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:21:20.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:20.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:21.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:22.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:22.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:22.200 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:21:22.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:21:22.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:22.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:22.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:23.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:23.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:23.696 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:21:23.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:21:23.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:23.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:24.593 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:25.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:25.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:25.595 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:21:25.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:21:25.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:25.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:26.072 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:27.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:27.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:27.074 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:21:27.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:21:27.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:27.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:27.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:28.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:28.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:28.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:21:28.985 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:21:28.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:28.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:29.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:30.776 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:30.776 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:30.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:21:30.777 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:21:30.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:30.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:31.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:32.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:32.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:32.354 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:21:32.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:21:32.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:32.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:32.825 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/136048,v1:127.0.0.1:6803/136048] [v2:127.0.0.1:6804/136048,v1:127.0.0.1:6805/136048] exists,up ce1d7a84-cfa4-42cd-ad32-b529b21c1077 2024-01-10T07:21:32.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:21:32.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:21:32.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:21:32.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-01-10T07:21:32.826 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-01-10T07:21:32.826 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-01-10T07:21:32.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-01-10T07:21:32.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-01-10T07:21:32.827 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=136048 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 136048 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-01-10T07:21:32.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-01-10T07:21:33.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:33.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 136048 2024-01-10T07:21:33.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-01-10T07:21:33.831 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-01-10T07:21:33.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-01-10T07:21:33.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-01-10T07:21:33.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-01-10T07:21:33.832 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-01-10T07:21:33.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=136048 2024-01-10T07:21:33.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-01-10T07:21:33.834 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-01-10T07:21:33.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-01-10T07:21:33.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:33.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 136048 2024-01-10T07:21:33.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-01-10T07:21:33.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-01-10T07:21:33.835 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-01-10T07:21:33.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:33.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 136048 2024-01-10T07:21:33.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-01-10T07:21:33.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-01-10T07:21:33.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 136048 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-01-10T07:21:34.136 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-01-10T07:21:34.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-01-10T07:21:34.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-01-10T07:21:34.138 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-01-10T07:21:34.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=135100 2024-01-10T07:21:34.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 135100 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-01-10T07:21:34.140 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 135100 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-01-10T07:21:34.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-01-10T07:21:34.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-01-10T07:21:34.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-01-10T07:21:34.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-01-10T07:21:34.245 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=135012 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 135012 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-01-10T07:21:34.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-01-10T07:21:34.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-01-10T07:21:34.346 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-01-10T07:21:34.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 135012 2024-01-10T07:21:34.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-01-10T07:21:34.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-01-10T07:21:34.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-01-10T07:21:34.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-01-10T07:21:39.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:21:39.351 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:21:39.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:21:39.356 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:21:39.356 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:21:39.357 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:21:39.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:21:39.358 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:21:39.358 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:21:39.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:21:39.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:21:39.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:21:39.361 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:21:39.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:21:39.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:21:39.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:21:39.368 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:21:39.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:39.369 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:39.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_kill_daemons td/ceph-helpers 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:21:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:21:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:21:39.373 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:21:39.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:21:39.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:21:39.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:21:39.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:21:39.375 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:21:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:21:39.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:21:39.378 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:21:39.378 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:21:39.379 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:21:39.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:21:39.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:21:39.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:21:39.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:39.381 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:39.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:21:39.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:21:39.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:21:39.383 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:21:39.384 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:21:39.384 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:39.384 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:39.384 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:21:39.386 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:21:39.386 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:21:39.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:21:39.737 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:21:39.737 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:21:39.737 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:21:39.737 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:21:39.737 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:39.738 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:39.738 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:39.738 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:21:40.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:21:40.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:21:40.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:21:40.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:40.008 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:21:40.009 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:21:40.009 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:40.120 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:21:40.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:21:40.121 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:21:40.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-01-10T07:21:40.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:21:40.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:21:40.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:21:40.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:21:40.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:21:40.222 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:21:41.116 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:21:41.116 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:21:41.116 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:21:41.117 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:21:41.117 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:41.117 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:41.117 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:41.117 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:21:41.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:21:41.145 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:21:41.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb334975-46ae-411f-b3a7-b49a091ce7d1 2024-01-10T07:21:41.147 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cb334975-46ae-411f-b3a7-b49a091ce7d1' 2024-01-10T07:21:41.148 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 cb334975-46ae-411f-b3a7-b49a091ce7d1 2024-01-10T07:21:41.148 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:21:41.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFRZ5lpt+dCRAAt2kapjsKsDFoxEGoC2yv9w== 2024-01-10T07:21:41.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFRZ5lpt+dCRAAt2kapjsKsDFoxEGoC2yv9w=="}' 2024-01-10T07:21:41.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb334975-46ae-411f-b3a7-b49a091ce7d1 -i td/ceph-helpers/0/new.json 2024-01-10T07:21:41.555 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:21:41.566 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:21:41.567 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFRZ5lpt+dCRAAt2kapjsKsDFoxEGoC2yv9w== --osd-uuid cb334975-46ae-411f-b3a7-b49a091ce7d1 2024-01-10T07:21:41.592 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.592+0000 7f5a9875b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:41.594 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.594+0000 7f5a9875b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:41.595 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.595+0000 7f5a9875b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:41.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.595+0000 7f5a9875b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:21:41.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.595+0000 7f5a9875b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:21:41.596 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.595+0000 7f5a9875b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:21:41.646 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:41.646+0000 7f5a9875b540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:21:47.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:21:47.194 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:21:47.194 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:21:47.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:21:47.195 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:21:47.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:21:47.485 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:21:47.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:21:47.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:21:47.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:21:47.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:21:47.508 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:47.508+0000 7fbbf39b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:47.523 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:47.523+0000 7fbbf39b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:47.524 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:47.524+0000 7fbbf39b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:21:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:21:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:21:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:21:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:21:47.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:21:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:21:47.957 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:21:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:47.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:48.060 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:48.060+0000 7fbbf39b2540 -1 Falling back to public interface 2024-01-10T07:21:48.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:49.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:49.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:49.458 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:21:49.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:21:49.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:49.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:49.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:50.586 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:21:50.586+0000 7fbbf39b2540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:21:50.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:50.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:50.944 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:21:50.944 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:21:50.945 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:50.945 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:51.571 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:52.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:52.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:52.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:21:52.573 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:21:52.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:52.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:53.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:54.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:54.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:54.033 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:21:54.033 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:21:54.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:54.034 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:54.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:55.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:55.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:55.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:21:55.580 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:21:55.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:55.580 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:56.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:57.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:57.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:57.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:21:57.050 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:21:57.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:57.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:57.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:21:58.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:21:58.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:21:58.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:21:58.628 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:21:58.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:21:58.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:21:59.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:00.087 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:00.087 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:00.087 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:22:00.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:22:00.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:00.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:00.680 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:01.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:01.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:01.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:22:01.682 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:22:01.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:01.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:02.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:03.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:03.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:03.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:22:03.579 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:22:03.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:03.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:04.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:05.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:05.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:05.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:22:05.507 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:22:05.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:05.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:06.114 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:07.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:07.116 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:07.116 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:22:07.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:22:07.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:07.117 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:08.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:09.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:09.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:09.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:22:09.093 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:22:09.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:09.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:09.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:10.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:22:10.992 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:22:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:10.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:11.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:12.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:12.902 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:12.903 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:22:12.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:22:12.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:12.903 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:13.406 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/138169,v1:127.0.0.1:6803/138169] [v2:127.0.0.1:6804/138169,v1:127.0.0.1:6805/138169] exists,up cb334975-46ae-411f-b3a7-b49a091ce7d1 2024-01-10T07:22:13.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:22:13.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:22:13.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:22:13.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-01-10T07:22:13.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:22:13.408 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:22:13.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:22:13.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:22:13.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:22:14.414 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:22:14.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:22:14.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-01-10T07:22:14.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:22:14.721 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:22:14.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:22:14.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:22:14.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:22:14.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:22:14.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-01-10T07:22:14.829 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:22:14.829 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:22:14.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:22:14.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:22:14.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:22:15.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:22:15.138 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-01-10T07:22:20.141 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:22:20.142 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:22:20.146 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:22:20.147 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:22:20.148 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:22:20.148 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:22:20.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:22:20.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:22:20.149 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:22:20.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:22:20.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:22:20.152 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:22:20.152 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:22:20.153 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:22:20.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:22:20.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:22:20.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:22:20.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:20.160 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:20.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_objectstore_tool td/ceph-helpers 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:22:20.162 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:22:20.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:22:20.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:22:20.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:22:20.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:22:20.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:22:20.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:22:20.165 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:22:20.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:22:20.166 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:22:20.166 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:22:20.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:22:20.168 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:22:20.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:22:20.169 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:22:20.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:22:20.170 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:22:20.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:22:20.172 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:22:20.172 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:20.172 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:20.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:22:20.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:22:20.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:22:20.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:22:20.174 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:22:20.174 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:20.175 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:20.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:22:20.177 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:22:20.519 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:22:20.519 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:22:20.519 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:22:20.519 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:22:20.519 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:20.520 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:20.520 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:22:20.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:22:20.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:22:20.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:22:20.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:22:20.773 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:22:20.773 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:22:20.774 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:22:20.775 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:22:20.886 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:20.887 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:20.887 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:22:20.887 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:22:20.887 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:22:20.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-01-10T07:22:20.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-01-10T07:22:20.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:22:20.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:22:20.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:22:20.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:22:20.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:22:20.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:21.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:22:21.884 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:22:21.884 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:22:21.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-01-10T07:22:21.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:22:21.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:22:21.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:22:21.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:22:21.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:22:21.913 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:22:21.914 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:22:21.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9e12e00-4377-41ff-bbdd-63354845d6a4 2024-01-10T07:22:21.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c9e12e00-4377-41ff-bbdd-63354845d6a4' 2024-01-10T07:22:21.915 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 c9e12e00-4377-41ff-bbdd-63354845d6a4 2024-01-10T07:22:21.916 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:22:21.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtRZ5lHoFdNxAAgbvDdyRCXXYXeDvW3VmgRg== 2024-01-10T07:22:21.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtRZ5lHoFdNxAAgbvDdyRCXXYXeDvW3VmgRg=="}' 2024-01-10T07:22:21.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9e12e00-4377-41ff-bbdd-63354845d6a4 -i td/ceph-helpers/0/new.json 2024-01-10T07:22:22.322 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:22:22.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:22:22.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtRZ5lHoFdNxAAgbvDdyRCXXYXeDvW3VmgRg== --osd-uuid c9e12e00-4377-41ff-bbdd-63354845d6a4 2024-01-10T07:22:22.359 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.359+0000 7f876bbc3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:22:22.361 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.361+0000 7f876bbc3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:22:22.362 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.362+0000 7f876bbc3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:22:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.362+0000 7f876bbc3540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:22:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.362+0000 7f876bbc3540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:22:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.362+0000 7f876bbc3540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:22:22.413 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:22.412+0000 7f876bbc3540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:22:27.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:22:27.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:22:27.733 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:22:27.733 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:22:27.734 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:22:27.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:22:27.977 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:22:27.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:22:27.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:22:27.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:22:27.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:22:28.001 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:28.000+0000 7fd3171b7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:22:28.003 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:28.002+0000 7fd3171b7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:22:28.004 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:28.003+0000 7fd3171b7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:28.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:22:28.249 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:22:28.249 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:28.249 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:28.527 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:28.526+0000 7fd3171b7540 -1 Falling back to public interface 2024-01-10T07:22:28.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:29.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:29.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:29.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:22:29.697 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:22:29.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:29.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:30.169 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:30.950 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:22:30.949+0000 7fd3171b7540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:22:31.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:31.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:31.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:22:31.171 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:22:31.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:31.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:31.712 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:32.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:32.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:32.714 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:22:32.714 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:22:32.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:32.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:33.173 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:34.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:34.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:34.175 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:22:34.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:22:34.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:34.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:34.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:35.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:35.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:35.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:22:35.708 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:22:35.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:35.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:36.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:37.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:37.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:37.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:22:37.176 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:22:37.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:37.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:37.706 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:38.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:38.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:38.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:22:38.708 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:22:38.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:38.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:39.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:40.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:40.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:40.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:22:40.185 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:22:40.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:40.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:40.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:41.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:41.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:41.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:22:41.694 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:22:41.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:41.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:42.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:43.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:43.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:43.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:22:43.240 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:22:43.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:43.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:43.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:44.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:44.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:44.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:22:44.953 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:22:44.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:44.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:45.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:46.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:46.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:46.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:22:46.730 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:22:46.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:46.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:47.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:48.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:22:48.232 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:22:48.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:48.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:48.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:49.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:49.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:49.857 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:22:49.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:22:49.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:49.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:50.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:51.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:51.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:51.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:22:51.704 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:22:51.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:51.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:52.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:22:53.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:22:53.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:22:53.266 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:22:53.266 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:22:53.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:22:53.267 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:22:53.738 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/140200,v1:127.0.0.1:6803/140200] [v2:127.0.0.1:6804/140200,v1:127.0.0.1:6805/140200] exists,up c9e12e00-4377-41ff-bbdd-63354845d6a4 2024-01-10T07:22:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:22:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:22:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:22:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-01-10T07:22:53.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:22:54.261 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:22:54.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:22:54.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:22:54.940 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:22:54.950 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:22:55.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:22:56.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-01-10T07:22:56.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:22:56.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:22:56.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:22:56.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:22:56.900 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:22:56.901 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:22:56.901 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:22:56.901 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:22:56.901 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:22:56.901 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:22:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:22:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:22:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:22:57.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:22:57.007 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:22:57.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:22:57.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:22:57.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:22:57.470 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:22:57.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836490 2024-01-10T07:22:57.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836490 2024-01-10T07:22:57.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836490' 2024-01-10T07:22:57.650 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:22:57.650 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836490 2024-01-10T07:22:57.650 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:22:57.650 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:22:57.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836490 2024-01-10T07:22:57.651 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:22:57.652 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836490 2024-01-10T07:22:57.652 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-01-10T07:22:57.652 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836490 2024-01-10T07:22:57.652 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:22:58.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836490 2024-01-10T07:22:58.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:22:59.135 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:22:59.135 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:22:59.607 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836490 -lt 21474836490 2024-01-10T07:22:59.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:22:59.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:22:59.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:23:00.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:23:00.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:23:00.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:23:00.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:23:00.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:23:00.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:23:00.226 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:23:00.226 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:23:00.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:23:00.694 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:23:00.694 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:23:00.694 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:23:01.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:23:01.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:23:01.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:23:01.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-01-10T07:23:01.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-01-10T07:23:01.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-01-10T07:23:01.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:23:01.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-01-10T07:23:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-01-10T07:23:02.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-01-10T07:23:03.925 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2024-01-10T07:23:03.925 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2024-01-10T07:23:03.925 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2024-01-10T07:23:03.925 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2024-01-10T07:23:04.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-01-10T07:23:04.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:23:04.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:23:04.202 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-01-10T07:23:04.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:23:04.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-01-10T07:23:04.204 INFO:tasks.workunit.client.0.smithi057.stderr:start osd.0 2024-01-10T07:23:04.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:23:04.204 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-01-10T07:23:04.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-01-10T07:23:04.204 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:23:04.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:23:04.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:23:04.227 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:04.227+0000 7f8f8d365540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:04.229 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:04.229+0000 7f8f8d365540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:04.231 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:04.230+0000 7f8f8d365540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:04.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-01-10T07:23:04.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:23:04.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:0 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:04.682 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:04.752 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:04.752+0000 7f8f8d365540 -1 Falling back to public interface 2024-01-10T07:23:05.223 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:06.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:06.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:06.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:23:06.225 INFO:tasks.workunit.client.0.smithi057.stderr:1 2024-01-10T07:23:06.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:06.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:06.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:07.392 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:07.391+0000 7f8f8d365540 -1 osd.0 14 log_to_monitors true 2024-01-10T07:23:07.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:07.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:23:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:2 2024-01-10T07:23:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:08.170 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 up in weight 1 up_from 17 up_thru 17 down_at 15 last_clean_interval [5,14) [v2:127.0.0.1:6802/141965,v1:127.0.0.1:6803/141965] [v2:127.0.0.1:6804/141965,v1:127.0.0.1:6805/141965] exists,up c9e12e00-4377-41ff-bbdd-63354845d6a4 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:23:08.171 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:23:08.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:23:08.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:23:08.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:23:08.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:23:08.262 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:23:08.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:23:08.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:23:08.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:23:08.754 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:23:08.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=73014444035 2024-01-10T07:23:08.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 73014444035 2024-01-10T07:23:08.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-73014444035' 2024-01-10T07:23:08.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:23:08.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-73014444035 2024-01-10T07:23:08.934 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:23:08.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:23:08.935 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-73014444035 2024-01-10T07:23:08.935 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:23:08.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=73014444035 2024-01-10T07:23:08.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 73014444035' 2024-01-10T07:23:08.936 INFO:tasks.workunit.client.0.smithi057.stderr:waiting osd.0 seq 73014444035 2024-01-10T07:23:08.936 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:23:09.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 1 -lt 73014444035 2024-01-10T07:23:09.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:23:10.406 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:23:10.406 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:23:10.881 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 73014444035 -lt 73014444035 2024-01-10T07:23:10.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:23:10.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:23:10.882 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:23:11.539 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:23:11.540 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:23:11.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:23:11.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:23:11.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:23:11.974 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-01-10T07:23:12.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:23:12.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:23:12.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:23:12.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-01-10T07:23:12.896 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-01-10T07:23:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-01-10T07:23:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-01-10T07:23:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-01-10T07:23:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-01-10T07:23:12.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-01-10T07:23:13.917 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2024-01-10T07:23:13.917 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2024-01-10T07:23:13.917 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2024-01-10T07:23:13.917 INFO:tasks.workunit.client.0.smithi057.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2024-01-10T07:23:13.917 INFO:tasks.workunit.client.0.smithi057.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-01-10T07:23:14.184 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:23:14.185 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:23:14.292 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:23:14.292 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:23:14.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:23:14.294 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:23:14.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:23:14.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:23:14.295 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:23:14.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:23:14.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:23:14.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:23:14.298 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:23:14.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:23:14.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:23:14.300 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:23:14.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:23:14.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:14.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:14.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:23:14.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:23:14.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:23:14.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:23:14.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_pg_scrub td/ceph-helpers 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_pg_scrub: local dir=td/ceph-helpers 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_pg_scrub: setup td/ceph-helpers 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:23:14.310 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:23:14.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:23:14.311 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:23:14.312 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:23:14.312 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:23:14.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:23:14.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:23:14.314 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:23:14.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:23:14.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:23:14.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:23:14.316 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:23:14.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:23:14.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:23:14.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:23:14.320 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:23:14.320 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:14.320 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:14.320 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:23:14.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:23:14.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:23:14.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:23:14.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:23:14.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:14.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:14.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:23:14.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:23:14.684 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:23:14.684 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:23:14.684 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:23:14.685 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:23:14.685 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:14.685 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:14.685 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:23:14.685 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:23:14.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:23:14.970 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:23:14.970 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:23:14.970 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:23:14.970 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:23:14.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:14.972 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:23:14.973 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:23:14.973 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:23:15.127 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:15.128 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:23:15.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:23:15.129 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:23:15.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_pg_scrub: run_mgr td/ceph-helpers x 2024-01-10T07:23:15.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:23:15.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:23:15.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:23:15.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:23:15.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:23:15.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:23:16.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:23:16.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_pg_scrub: run_osd td/ceph-helpers 0 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:23:16.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:23:16.161 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:23:16.162 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:23:16.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:23:16.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:23:16.163 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:23:16.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8d7c10f-51fc-4d78-b9b2-f3ff8d21ff18 2024-01-10T07:23:16.163 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 b8d7c10f-51fc-4d78-b9b2-f3ff8d21ff18 2024-01-10T07:23:16.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b8d7c10f-51fc-4d78-b9b2-f3ff8d21ff18' 2024-01-10T07:23:16.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:23:16.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkRZ5lvnCLChAAVC/K2bC//OHBbgDqJEU31Q== 2024-01-10T07:23:16.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkRZ5lvnCLChAAVC/K2bC//OHBbgDqJEU31Q=="}' 2024-01-10T07:23:16.178 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8d7c10f-51fc-4d78-b9b2-f3ff8d21ff18 -i td/ceph-helpers/0/new.json 2024-01-10T07:23:16.578 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:23:16.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:23:16.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkRZ5lvnCLChAAVC/K2bC//OHBbgDqJEU31Q== --osd-uuid b8d7c10f-51fc-4d78-b9b2-f3ff8d21ff18 2024-01-10T07:23:16.616 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.616+0000 7f1433a72540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:16.618 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.618+0000 7f1433a72540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:16.619 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.619+0000 7f1433a72540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:16.619 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.619+0000 7f1433a72540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:23:16.619 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.619+0000 7f1433a72540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:23:16.619 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.619+0000 7f1433a72540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:23:16.691 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:16.691+0000 7f1433a72540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:23:21.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:23:21.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:23:21.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:23:21.981 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:23:21.981 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:23:22.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:23:22.224 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:23:22.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:23:22.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:23:22.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:23:22.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:23:22.248 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:22.248+0000 7f8ca52a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:22.250 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:22.250+0000 7f8ca52a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:22.251 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:22.251+0000 7f8ca52a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:23:22.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:23:22.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:22.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:22.771 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:22.771+0000 7f8ca52a9540 -1 Falling back to public interface 2024-01-10T07:23:23.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:24.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:24.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:24.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:23:24.018 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:23:24.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:24.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:24.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:25.165 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:23:25.164+0000 7f8ca52a9540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:23:25.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:25.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:25.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:23:25.494 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:23:25.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:25.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:25.953 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:26.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:26.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:26.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:23:26.955 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:23:26.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:26.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:27.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:28.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:28.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:28.421 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:23:28.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:23:28.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:28.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:28.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:29.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:29.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:29.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:23:29.926 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:23:29.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:29.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:30.389 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:31.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:31.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:31.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:23:31.391 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:23:31.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:31.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:31.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:32.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:32.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:32.910 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:23:32.910 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:23:32.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:32.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:33.389 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:34.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:34.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:34.391 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:23:34.391 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:23:34.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:34.392 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:34.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:35.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:35.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:35.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:23:35.892 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:23:35.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:35.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:36.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:37.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:37.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:37.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:23:37.458 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:23:37.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:37.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:38.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:39.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:39.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:39.047 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:23:39.048 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:23:39.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:39.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:39.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:40.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:40.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:40.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:23:40.925 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:23:40.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:40.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:41.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:42.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:42.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:42.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:23:42.720 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:23:42.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:42.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:43.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:44.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:44.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:44.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:23:44.462 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:23:44.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:44.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:45.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:46.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:46.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:46.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:23:46.576 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:23:46.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:46.576 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:47.058 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:23:48.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:23:48.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:23:48.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:23:48.060 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:23:48.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:23:48.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:23:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/144275,v1:127.0.0.1:6803/144275] [v2:127.0.0.1:6804/144275,v1:127.0.0.1:6805/144275] exists,up b8d7c10f-51fc-4d78-b9b2-f3ff8d21ff18 2024-01-10T07:23:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:23:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:23:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:23:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_pg_scrub: create_rbd_pool 2024-01-10T07:23:48.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:23:48.996 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:23:49.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:23:49.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:23:49.774 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:23:49.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:23:50.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:23:51.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_pg_scrub: wait_for_clean 2024-01-10T07:23:51.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:23:51.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:23:51.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:23:51.783 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:23:51.783 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:23:51.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:23:51.784 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:23:51.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:23:51.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:23:51.784 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:23:51.888 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:23:51.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:23:51.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:23:51.889 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:23:51.889 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:23:52.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:23:52.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:23:52.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:23:52.353 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:23:52.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836486 2024-01-10T07:23:52.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836486 2024-01-10T07:23:52.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836486' 2024-01-10T07:23:52.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:23:52.514 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836486 2024-01-10T07:23:52.514 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:23:52.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:23:52.515 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836486 2024-01-10T07:23:52.515 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:23:52.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836486 2024-01-10T07:23:52.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-01-10T07:23:52.516 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836486 2024-01-10T07:23:52.516 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:23:52.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836484 -lt 21474836486 2024-01-10T07:23:52.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:23:53.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:23:53.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:23:54.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836484 -lt 21474836486 2024-01-10T07:23:54.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:23:55.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2024-01-10T07:23:55.474 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:23:55.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836486 2024-01-10T07:23:55.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:23:55.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:23:55.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:23:56.582 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:23:56.583 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:23:57.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:23:57.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:23:57.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:23:57.014 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_pg_scrub: pg_scrub 1.0 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: pg_scrub: local pgid=1.0 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: get_last_scrub_stamp 1.0 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:23:57.629 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:23:58.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: local last_scrub=2024-01-10T07:23:49.404907+0000 2024-01-10T07:23:58.098 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: pg_scrub: ceph pg scrub 1.0 2024-01-10T07:23:58.533 INFO:tasks.workunit.client.0.smithi057.stderr:instructing pg 1.0 on osd.0 to scrub 2024-01-10T07:23:58.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: pg_scrub: wait_for_scrub 1.0 2024-01-10T07:23:49.404907+0000 2024-01-10T07:23:58.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2024-01-10T07:23:58.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2024-01-10T07:23:49.404907+0000 2024-01-10T07:23:58.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2024-01-10T07:23:58.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2024-01-10T07:23:58.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:23:58.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:23:58.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:23:58.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:23:58.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:23:58.544 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:23:58.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:23:49.404907+0000 '>' 2024-01-10T07:23:49.404907+0000 2024-01-10T07:23:58.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:23:59.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:23:59.977 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:23:59.977 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:23:59.977 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:23:59.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:23:59.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:23:59.978 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:00.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:23:49.404907+0000 '>' 2024-01-10T07:23:49.404907+0000 2024-01-10T07:24:00.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:01.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:01.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:24:01.427 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:01.427 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:01.427 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:01.427 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:01.427 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:01.879 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:23:49.404907+0000 '>' 2024-01-10T07:23:49.404907+0000 2024-01-10T07:24:01.879 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:02.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:02.880 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:24:02.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:02.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:02.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:02.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:02.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:03.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:23:49.404907+0000 '>' 2024-01-10T07:23:49.404907+0000 2024-01-10T07:24:03.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:04.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:04.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:24:04.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:04.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:04.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:04.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:04.331 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:04.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:24:02.484074+0000 '>' 2024-01-10T07:23:49.404907+0000 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_for_scrub: return 0 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:04.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_pg_scrub: TIMEOUT=1 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_pg_scrub: pg_scrub 1.0 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: pg_scrub: local pgid=1.0 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: get_last_scrub_stamp 1.0 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:04.892 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:05.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: local last_scrub=2024-01-10T07:24:02.484074+0000 2024-01-10T07:24:05.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: pg_scrub: ceph pg scrub 1.0 2024-01-10T07:24:05.760 INFO:tasks.workunit.client.0.smithi057.stderr:instructing pg 1.0 on osd.0 to scrub 2024-01-10T07:24:05.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: pg_scrub: wait_for_scrub 1.0 2024-01-10T07:24:02.484074+0000 2024-01-10T07:24:05.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2024-01-10T07:24:05.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2024-01-10T07:24:02.484074+0000 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:05.772 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:06.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:24:02.484074+0000 '>' 2024-01-10T07:24:02.484074+0000 2024-01-10T07:24:06.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: wait_for_scrub: return 1 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1874: test_pg_scrub: teardown td/ceph-helpers 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:07.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:07.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:07.317 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:07.317 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:24:07.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:24:07.318 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:24:07.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:24:07.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:24:07.320 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:24:07.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:07.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:24:07.322 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:24:07.322 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:07.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:24:07.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:24:07.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:24:07.330 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:24:07.330 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:07.330 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:07.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:24:07.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:24:07.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:24:07.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_repair td/ceph-helpers 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1823: test_repair: local dir=td/ceph-helpers 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1825: test_repair: setup td/ceph-helpers 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:07.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:07.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:07.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:07.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:07.334 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:24:07.335 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:24:07.336 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:24:07.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:24:07.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:24:07.337 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:24:07.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:07.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:24:07.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:24:07.339 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:07.340 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:24:07.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:24:07.341 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:24:07.343 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:24:07.343 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:07.343 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:07.343 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:24:07.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:24:07.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:24:07.344 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:24:07.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:24:07.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:07.345 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:07.345 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1826: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:24:07.347 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:24:07.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:24:07.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:24:07.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:24:07.348 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:24:07.702 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:07.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:24:07.963 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:24:07.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:24:07.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:24:07.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:24:07.964 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:24:07.965 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:24:07.965 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:24:07.965 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:24:07.966 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:24:08.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:24:08.078 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:24:08.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:24:08.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:24:08.079 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:24:08.079 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:24:08.079 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:24:08.079 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:24:08.080 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:24:08.080 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:08.080 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:08.080 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:24:08.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:24:08.080 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:24:08.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1827: test_repair: run_mgr td/ceph-helpers x 2024-01-10T07:24:08.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:24:08.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:24:08.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:24:08.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:24:08.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:24:08.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:24:09.073 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:24:09.102 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1828: test_repair: run_osd td/ceph-helpers 0 2024-01-10T07:24:09.102 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:24:09.102 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:24:09.102 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:24:09.102 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:24:09.102 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:24:09.103 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:24:09.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:24:09.105 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:24:09.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=afdf6b67-74de-49d5-ba40-fe5dced7057f 2024-01-10T07:24:09.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 afdf6b67-74de-49d5-ba40-fe5dced7057f' 2024-01-10T07:24:09.105 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 afdf6b67-74de-49d5-ba40-fe5dced7057f 2024-01-10T07:24:09.106 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:24:09.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZRp5lFxUVBxAAha/8okhGOrRMvOIywd9/VA== 2024-01-10T07:24:09.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZRp5lFxUVBxAAha/8okhGOrRMvOIywd9/VA=="}' 2024-01-10T07:24:09.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new afdf6b67-74de-49d5-ba40-fe5dced7057f -i td/ceph-helpers/0/new.json 2024-01-10T07:24:09.513 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:24:09.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:24:09.525 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZRp5lFxUVBxAAha/8okhGOrRMvOIywd9/VA== --osd-uuid afdf6b67-74de-49d5-ba40-fe5dced7057f 2024-01-10T07:24:09.550 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.550+0000 7fbdb1a99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:09.552 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.552+0000 7fbdb1a99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:09.553 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.553+0000 7fbdb1a99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:09.553 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.553+0000 7fbdb1a99540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:24:09.553 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.553+0000 7fbdb1a99540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:24:09.553 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.553+0000 7fbdb1a99540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:24:09.617 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:09.616+0000 7fbdb1a99540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:24:14.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:24:14.915 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:24:14.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:24:14.916 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:24:14.916 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:24:15.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:24:15.160 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:24:15.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:24:15.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:24:15.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:24:15.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:24:15.184 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:15.184+0000 7f551dc26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:15.187 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:15.187+0000 7f551dc26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:15.188 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:15.188+0000 7f551dc26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:24:15.428 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:24:15.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:15.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:15.709 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:15.709+0000 7f551dc26540 -1 Falling back to public interface 2024-01-10T07:24:15.939 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:16.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:16.941 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:16.941 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:24:16.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:24:16.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:16.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:17.415 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:18.120 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:18.120+0000 7f551dc26540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:24:18.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:18.416 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:18.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:24:18.417 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:24:18.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:18.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:18.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:19.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:19.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:19.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:24:19.930 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:24:19.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:19.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:20.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:21.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:21.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:21.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:24:21.397 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:24:21.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:21.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:21.897 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:22.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:22.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:22.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:24:22.899 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:24:22.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:22.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:23.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:24.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:24.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:24.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:24:24.500 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:24:24.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:24.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:25.017 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:26.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:26.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:26.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:24:26.019 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:24:26.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:26.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:26.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:27.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:27.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:27.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:24:27.513 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:24:27.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:27.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:27.998 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:29.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:29.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:29.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:24:29.000 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:24:29.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:29.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:29.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:30.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:30.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:30.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:24:30.459 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:24:30.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:30.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:31.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:32.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:32.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:32.126 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:24:32.127 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:24:32.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:32.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:33.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:34.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:34.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:34.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:24:34.130 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:24:34.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:34.130 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:34.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:35.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:35.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:35.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:24:35.641 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:24:35.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:35.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:36.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:37.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:37.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:37.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:24:37.450 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:24:37.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:37.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:38.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:24:39.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:24:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:24:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:24:39.372 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:24:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:24:39.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:24:39.873 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/146983,v1:127.0.0.1:6803/146983] [v2:127.0.0.1:6804/146983,v1:127.0.0.1:6805/146983] exists,up afdf6b67-74de-49d5-ba40-fe5dced7057f 2024-01-10T07:24:39.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:24:39.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:24:39.874 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:24:39.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: test_repair: create_rbd_pool 2024-01-10T07:24:39.875 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:24:40.365 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:24:40.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:24:40.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:24:41.041 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:24:41.051 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:24:42.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:24:43.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: test_repair: wait_for_clean 2024-01-10T07:24:44.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:24:44.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:24:44.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:24:44.321 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:24:44.324 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:24:44.325 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836487 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836487 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836487' 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836487 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:24:44.326 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836487 2024-01-10T07:24:44.327 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:24:44.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836487 2024-01-10T07:24:44.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-01-10T07:24:44.327 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836487 2024-01-10T07:24:44.327 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:24:44.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836487 2024-01-10T07:24:44.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:24:45.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:24:45.548 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:24:46.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836487 2024-01-10T07:24:46.024 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:24:46.024 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:24:46.024 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:24:46.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:24:46.678 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:24:47.143 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:24:47.143 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:24:47.144 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:24:47.144 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: test_repair: repair 1.0 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: repair: local pgid=1.0 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: get_last_scrub_stamp 1.0 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:47.782 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:48.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: local last_scrub=2024-01-10T07:24:40.773456+0000 2024-01-10T07:24:48.261 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: repair: ceph pg repair 1.0 2024-01-10T07:24:48.685 INFO:tasks.workunit.client.0.smithi057.stderr:instructing pg 1.0 on osd.0 to repair 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1819: repair: wait_for_scrub 1.0 2024-01-10T07:24:40.773456+0000 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2024-01-10T07:24:40.773456+0000 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:48.696 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:49.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:24:40.773456+0000 '>' 2024-01-10T07:24:40.773456+0000 2024-01-10T07:24:49.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:50.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:50.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:24:50.208 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:50.208 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:50.208 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:50.208 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:50.208 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:50.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:24:40.773456+0000 '>' 2024-01-10T07:24:40.773456+0000 2024-01-10T07:24:50.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:51.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:51.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2024-01-10T07:24:51.663 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:51.663 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:51.663 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:51.664 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:51.664 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:52.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:24:50.058427+0000 '>' 2024-01-10T07:24:40.773456+0000 2024-01-10T07:24:52.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_for_scrub: return 0 2024-01-10T07:24:52.112 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-01-10T07:24:52.112 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:52.113 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:52.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:52.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:52.113 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: test_repair: TIMEOUT=1 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: test_repair: repair 1.0 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: repair: local pgid=1.0 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: get_last_scrub_stamp 1.0 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:52.218 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:52.653 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: local last_scrub=2024-01-10T07:24:50.058427+0000 2024-01-10T07:24:52.653 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: repair: ceph pg repair 1.0 2024-01-10T07:24:53.084 INFO:tasks.workunit.client.0.smithi057.stderr:instructing pg 1.0 on osd.0 to repair 2024-01-10T07:24:53.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1819: repair: wait_for_scrub 1.0 2024-01-10T07:24:50.058427+0000 2024-01-10T07:24:53.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2024-01-10T07:24:53.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2024-01-10T07:24:50.058427+0000 2024-01-10T07:24:53.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2024-01-10T07:24:53.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2024-01-10T07:24:53.096 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2024-01-10T07:24:53.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-01-10T07:24:53.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-01-10T07:24:53.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-01-10T07:24:53.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-01-10T07:24:53.096 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-01-10T07:24:53.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2024-01-10T07:24:50.058427+0000 '>' 2024-01-10T07:24:50.058427+0000 2024-01-10T07:24:53.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2024-01-10T07:24:54.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2024-01-10T07:24:54.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2024-01-10T07:24:54.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: wait_for_scrub: return 1 2024-01-10T07:24:54.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_repair: teardown td/ceph-helpers 2024-01-10T07:24:54.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:24:54.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:24:54.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:24:54.535 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:54.535 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:54.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:54.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:54.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:54.641 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:54.642 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:24:54.642 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:24:54.643 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:24:54.644 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:24:54.644 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:24:54.645 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:24:54.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:54.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:24:54.647 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:24:54.648 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:54.648 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:24:54.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:24:54.649 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:24:54.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:24:54.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:54.656 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:54.656 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:24:54.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:24:54.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:24:54.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:24:54.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_run_in_background td/ceph-helpers 2024-01-10T07:24:54.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: test_run_in_background: local pids 2024-01-10T07:24:54.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_run_in_background: run_in_background pids sleep 1 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 148524"' 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 148524' 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_run_in_background: run_in_background pids sleep 1 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sleep 1 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 148526"' 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 148526' 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/148527: /' 2024-01-10T07:24:54.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sleep 1 2024-01-10T07:24:54.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/148530: /' 2024-01-10T07:24:54.661 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_run_in_background: echo 148524 148526 2024-01-10T07:24:54.661 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_run_in_background: wc -w 2024-01-10T07:24:54.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_run_in_background: test 2 = 2 2024-01-10T07:24:54.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: test_run_in_background: wait 148524 148526 2024-01-10T07:24:55.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_run_mon td/ceph-helpers 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:24:55.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:55.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:55.663 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:24:55.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:24:55.664 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:24:55.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:24:55.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:24:55.665 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:24:55.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:55.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:24:55.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:24:55.668 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:24:55.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:24:55.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:24:55.671 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:24:55.672 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:24:55.672 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:55.672 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:55.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:24:55.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:24:55.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:24:55.673 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:24:55.675 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:24:55.675 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:55.675 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:55.675 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:24:55.677 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:24:55.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:24:55.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:24:55.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:24:55.677 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-01-10T07:24:55.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:24:55.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:24:55.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:24:55.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:24:55.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:24:55.678 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:24:56.051 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:56.052 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:24:56.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:24:56.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:24:56.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:24:56.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:24:56.305 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:24:56.306 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:24:56.306 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:24:56.306 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:24:56.306 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:24:56.306 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:24:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:24:56.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:24:56.307 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:24:56.419 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:24:56.420 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:56.421 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:56.421 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:24:56.421 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:24:56.421 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:24:56.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-01-10T07:24:56.521 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-01-10T07:24:57.385 INFO:tasks.workunit.client.0.smithi057.stderr:dumped monmap epoch 1 2024-01-10T07:24:57.395 INFO:tasks.workunit.client.0.smithi057.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-01-10T07:24:57.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-01-10T07:24:57.396 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:24:57.396 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:24:57.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:24:57.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:24:57.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:24:57.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:24:57.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-01-10T07:24:57.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:24:57.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:24:57.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:24:57.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:24:57.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:24:57.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-01-10T07:24:57.520 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:57.520+0000 7fd57dd57b80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-01-10T07:24:57.522 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:57.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:24:57.777 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:24:57.778 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:24:57.779 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:24:57.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:24:57.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:24:57.881 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:24:57.882 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:24:57.882 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:24:57.882 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:24:57.882 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:24:57.882 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:24:57.883 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:24:57.883 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:57.883 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:57.883 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:24:57.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:24:57.883 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:24:57.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-01-10T07:24:57.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:24:57.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:24:57.984 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:24:57.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7e7ed25-48e2-40d3-be1a-79c583a7f970 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 e7e7ed25-48e2-40d3-be1a-79c583a7f970 2024-01-10T07:24:57.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e7e7ed25-48e2-40d3-be1a-79c583a7f970' 2024-01-10T07:24:57.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:24:57.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJRp5lrHaOOxAAOW6pPCNgJWV6AxkyhjoJoA== 2024-01-10T07:24:57.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJRp5lrHaOOxAAOW6pPCNgJWV6AxkyhjoJoA=="}' 2024-01-10T07:24:57.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7e7ed25-48e2-40d3-be1a-79c583a7f970 -i td/ceph-helpers/0/new.json 2024-01-10T07:24:58.356 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:24:58.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:24:58.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJRp5lrHaOOxAAOW6pPCNgJWV6AxkyhjoJoA== --osd-uuid e7e7ed25-48e2-40d3-be1a-79c583a7f970 2024-01-10T07:24:58.392 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.392+0000 7fd64149b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:58.394 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.394+0000 7fd64149b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:58.395 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.395+0000 7fd64149b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:24:58.395 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.395+0000 7fd64149b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:24:58.395 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.395+0000 7fd64149b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:24:58.396 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.395+0000 7fd64149b540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:24:58.472 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:24:58.472+0000 7fd64149b540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:25:03.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:25:03.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:25:03.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:25:03.848 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:25:03.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:25:04.099 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:25:04.099 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:25:04.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:25:04.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:25:04.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:25:04.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:25:04.122 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:04.122+0000 7fd3ff18b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:04.124 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:04.124+0000 7fd3ff18b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:04.125 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:04.125+0000 7fd3ff18b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:04.330 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:25:04.331 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:25:04.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:04.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:04.560 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:04.650 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:04.650+0000 7fd3ff18b540 -1 Falling back to public interface 2024-01-10T07:25:05.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:05.562 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:05.562 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:25:05.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:25:05.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:05.563 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:05.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:06.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:06.806 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:06.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:25:06.807 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:25:06.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:06.807 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:07.001 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:07.001+0000 7fd3ff18b540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:25:07.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:08.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:08.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:08.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:25:08.050 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:25:08.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:08.050 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:08.332 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:09.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:09.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:09.333 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:25:09.334 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:25:09.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:09.334 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:09.577 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:10.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:10.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:10.578 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:25:10.578 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:25:10.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:10.579 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:10.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:11.822 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:11.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:11.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:25:11.823 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:25:11.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:11.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:12.065 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:13.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:13.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:13.066 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:25:13.067 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:25:13.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:13.067 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:13.311 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:14.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:14.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:14.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:25:14.313 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:25:14.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:14.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:14.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:15.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:15.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:15.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:25:15.554 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:25:15.555 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:15.555 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:15.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:16.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:16.801 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:16.801 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:25:16.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:25:16.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:16.802 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:17.043 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:18.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:18.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:18.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:25:18.045 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:25:18.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:18.046 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:18.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:19.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:19.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:19.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:25:19.287 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:25:19.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:19.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:19.528 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:20.530 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:20.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:20.531 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:25:20.531 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:25:20.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:20.532 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:20.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:21.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:21.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:21.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:25:21.773 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:25:21.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:21.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:22.014 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:23.015 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:23.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:23.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:25:23.016 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:25:23.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:23.016 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:23.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:24.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:24.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:24.366 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:25:24.367 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:25:24.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:24.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:24.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:25.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:25.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:25.611 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:25:25.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:25:25.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:25.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:25.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:26.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:26.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:26.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:25:26.855 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:25:26.855 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:26.856 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:25:27.207 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/149564,v1:127.0.0.1:6801/149564] [v2:127.0.0.1:6802/149564,v1:127.0.0.1:6803/149564] exists,up e7e7ed25-48e2-40d3-be1a-79c583a7f970 2024-01-10T07:25:27.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:25:27.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:25:27.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:25:27.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-01-10T07:25:27.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:25:27.212 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:25:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61d9ca72-7f96-46e7-9f60-906935829e26 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 61d9ca72-7f96-46e7-9f60-906935829e26' 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 61d9ca72-7f96-46e7-9f60-906935829e26 2024-01-10T07:25:27.214 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:25:27.226 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnRp5lh7CADRAA2Ta891y2W4aafluGeyOVgQ== 2024-01-10T07:25:27.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnRp5lh7CADRAA2Ta891y2W4aafluGeyOVgQ=="}' 2024-01-10T07:25:27.227 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61d9ca72-7f96-46e7-9f60-906935829e26 -i td/ceph-helpers/1/new.json 2024-01-10T07:25:27.577 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:25:27.587 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:25:27.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnRp5lh7CADRAA2Ta891y2W4aafluGeyOVgQ== --osd-uuid 61d9ca72-7f96-46e7-9f60-906935829e26 2024-01-10T07:25:27.614 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.613+0000 7f6685f57540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:27.657 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.657+0000 7f6685f57540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:27.658 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.658+0000 7f6685f57540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:27.659 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.658+0000 7f6685f57540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:25:27.659 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.658+0000 7f6685f57540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:25:27.659 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.658+0000 7f6685f57540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:25:27.698 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:27.697+0000 7f6685f57540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:25:33.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:25:33.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:25:33.504 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:25:33.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:25:33.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:25:33.755 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:25:33.755 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:25:33.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:25:33.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:25:33.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:25:33.756 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:25:33.778 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:33.778+0000 7fd52d6d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:33.780 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:33.780+0000 7fd52d6d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:33.781 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:33.781+0000 7fd52d6d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:33.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:25:33.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:25:33.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:25:33.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:25:33.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:25:33.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:33.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:25:33.987 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:25:33.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:33.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:34.215 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:34.297 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:34.297+0000 7fd52d6d3540 -1 Falling back to public interface 2024-01-10T07:25:35.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:35.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:35.217 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:25:35.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:25:35.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:35.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:35.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:36.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:36.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:36.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:25:36.461 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:25:36.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:36.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:36.661 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:36.661+0000 7fd52d6d3540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:25:36.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:37.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:37.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:37.759 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:25:37.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:25:37.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:37.760 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:37.998 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:38.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:38.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:38.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:25:39.000 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:25:39.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:39.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:39.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:40.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:40.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:40.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:25:40.242 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:25:40.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:40.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:40.484 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:41.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:41.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:41.485 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:25:41.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:25:41.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:41.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:41.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:42.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:42.730 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:42.731 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:25:42.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:25:42.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:42.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:42.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:43.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:43.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:43.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:25:43.975 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:25:43.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:43.976 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:44.219 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:45.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:45.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:45.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:25:45.221 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:25:45.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:45.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:45.463 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:46.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:46.464 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:46.464 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:25:46.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:25:46.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:46.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:46.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:47.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:47.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:47.709 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:25:47.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:25:47.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:47.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:47.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:48.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:48.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:48.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:25:48.954 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:25:48.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:48.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:49.196 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:50.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:50.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:50.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:25:50.198 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:25:50.198 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:50.199 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:50.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:51.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:51.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:51.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:25:51.442 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:25:51.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:51.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:51.685 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:52.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:52.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:52.687 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:25:52.687 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:25:52.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:52.688 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:52.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:53.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:53.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:53.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:25:53.933 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:25:53.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:53.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:54.174 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:55.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:55.175 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:55.176 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:25:55.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:25:55.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:55.176 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:55.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:56.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:56.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:56.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:25:56.419 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:25:56.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:56.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:56.681 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:25:57.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:25:57.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:25:57.683 INFO:tasks.workunit.client.0.smithi057.stdout:19 2024-01-10T07:25:57.683 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-01-10T07:25:57.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:25:57.684 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:25:57.925 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/151365,v1:127.0.0.1:6809/151365] [v2:127.0.0.1:6810/151365,v1:127.0.0.1:6811/151365] exists,up 61d9ca72-7f96-46e7-9f60-906935829e26 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:25:57.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:25:57.931 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:25:57.932 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8dc79851-f50a-4bbc-9134-d1fbe73b94e5 2024-01-10T07:25:57.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8dc79851-f50a-4bbc-9134-d1fbe73b94e5' 2024-01-10T07:25:57.933 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 8dc79851-f50a-4bbc-9134-d1fbe73b94e5 2024-01-10T07:25:57.933 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:25:57.945 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFRp5lglhTOBAARl+YfcypyajPadRUVGRO8w== 2024-01-10T07:25:57.945 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFRp5lglhTOBAARl+YfcypyajPadRUVGRO8w=="}' 2024-01-10T07:25:57.945 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8dc79851-f50a-4bbc-9134-d1fbe73b94e5 -i td/ceph-helpers/2/new.json 2024-01-10T07:25:58.271 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:25:58.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-01-10T07:25:58.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFRp5lglhTOBAARl+YfcypyajPadRUVGRO8w== --osd-uuid 8dc79851-f50a-4bbc-9134-d1fbe73b94e5 2024-01-10T07:25:58.308 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.308+0000 7f4793da1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:58.344 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.343+0000 7f4793da1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:58.344 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.344+0000 7f4793da1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:25:58.345 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.345+0000 7f4793da1540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:25:58.345 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.345+0000 7f4793da1540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:25:58.345 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.345+0000 7f4793da1540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:25:58.384 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:25:58.384+0000 7f4793da1540 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-01-10T07:26:03.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-01-10T07:26:03.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:26:03.928 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T07:26:03.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T07:26:03.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:26:04.205 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T07:26:04.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T07:26:04.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:26:04.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:26:04.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:26:04.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:26:04.228 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:04.228+0000 7f7d75d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:04.263 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:04.262+0000 7f7d75d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:04.263 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:04.263+0000 7f7d75d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T07:26:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:26:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T07:26:04.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:26:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:26:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:26:04.474 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:26:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:04.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:04.702 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:04.783 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:04.783+0000 7f7d75d55540 -1 Falling back to public interface 2024-01-10T07:26:05.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:05.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:05.704 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:26:05.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:26:05.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:05.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:05.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:06.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:06.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:06.948 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:26:06.948 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:26:06.949 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:06.949 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:07.172 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:07.171+0000 7f7d75d55540 -1 osd.2 0 log_to_monitors true 2024-01-10T07:26:07.249 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:08.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:08.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:08.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:26:08.251 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:26:08.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:08.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:08.493 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:09.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:09.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:09.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:26:09.497 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:26:09.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:09.498 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:09.738 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:10.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:10.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:10.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:26:10.740 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:26:10.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:10.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:10.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:11.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:11.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:11.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:26:11.987 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:26:11.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:11.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:12.231 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:13.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:13.232 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:13.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:26:13.233 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:26:13.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:13.233 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:13.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:14.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:14.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:14.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:26:14.475 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:26:14.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:14.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:14.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:15.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:15.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:15.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:26:15.722 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:26:15.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:15.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:15.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:16.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:16.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:16.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:26:16.966 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:26:16.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:16.966 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:17.209 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:18.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:18.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:18.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:26:18.211 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:26:18.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:18.211 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:18.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:19.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:19.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:19.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:26:19.455 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:26:19.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:19.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:19.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:20.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:20.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:20.698 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:26:20.698 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:26:20.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:20.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:20.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:21.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:21.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:21.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:26:21.937 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:26:21.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:21.937 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:22.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:23.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:23.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:23.181 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:26:23.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:26:23.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:23.182 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:23.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:24.424 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:24.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:24.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:26:24.425 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:26:24.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:24.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:24.664 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:25.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:25.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:25.666 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:26:25.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:26:25.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:25.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:25.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:26.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:26.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:26.911 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:26:26.911 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:26:26.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:26.912 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:27.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:28.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:28.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:28.212 INFO:tasks.workunit.client.0.smithi057.stdout:19 2024-01-10T07:26:28.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-01-10T07:26:28.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:28.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:26:28.464 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/153200,v1:127.0.0.1:6817/153200] [v2:127.0.0.1:6818/153200,v1:127.0.0.1:6819/153200] exists,up 8dc79851-f50a-4bbc-9134-d1fbe73b94e5 2024-01-10T07:26:28.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:26:28.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:26:28.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:26:28.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-01-10T07:26:28.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:26:28.696 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:26:28.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:26:28.707 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:26:29.191 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:26:29.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:26:30.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:26:31.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-01-10T07:26:31.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-01-10T07:26:32.063 INFO:tasks.workunit.client.0.smithi057.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 23 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-01-10T07:26:32.065 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get osd_pool_default_size 2024-01-10T07:26:32.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-01-10T07:26:32.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-01-10T07:26:32.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-01-10T07:26:32.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-01-10T07:26:32.255 INFO:tasks.workunit.client.0.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-01-10T07:26:32.255 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.254+0000 7f59659ce700 -1 failed for service _ceph-mon._tcp 2024-01-10T07:26:32.255 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.254+0000 7f59659ce700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-01-10T07:26:32.255 INFO:tasks.workunit.client.0.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-01-10T07:26:32.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-01-10T07:26:32.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-01-10T07:26:32.346 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.346+0000 7f6597f76700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-01-10T07:26:32.346 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.346+0000 7f6597f76700 -1 AuthRegistry(0x7f6590064610) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-01-10T07:26:32.347 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.346+0000 7f6597f76700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-01-10T07:26:32.347 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.346+0000 7f6597f76700 -1 AuthRegistry(0x7f6597f74ea0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-01-10T07:26:32.348 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.348+0000 7f6597f76700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-01-10T07:26:32.348 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.348+0000 7f6597f76700 -1 AuthRegistry(0x7f6590061bb0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-01-10T07:26:32.348 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.348+0000 7f6597f76700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-01-10T07:26:32.348 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.348+0000 7f6597f76700 -1 AuthRegistry(0x7f65900fd700) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-01-10T07:26:32.348 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.348+0000 7f6597f76700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-01-10T07:26:32.348 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:32.348+0000 7f6597f76700 -1 AuthRegistry(0x7f6590061bb0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-01-10T07:26:32.505 INFO:tasks.workunit.client.0.smithi057.stdout: cluster: 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: id: c182505b-1213-4a65-a0b1-c14e269ca5fa 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: health: HEALTH_OK 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: services: 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: mon: 1 daemons, quorum a (age 94s) 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: mgr: no daemons active 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: osd: 3 osds: 3 up (since 5s), 3 in (since 34s) 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: data: 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: pools: 0 pools, 0 pgs 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: objects: 0 objects, 0 B 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: usage: 0 B used, 0 B / 0 B avail 2024-01-10T07:26:32.506 INFO:tasks.workunit.client.0.smithi057.stdout: pgs: 2024-01-10T07:26:32.507 INFO:tasks.workunit.client.0.smithi057.stdout: 2024-01-10T07:26:32.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-01-10T07:26:32.516 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:26:32.516 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:26:32.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:26:32.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:26:32.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:26:37.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:26:37.829 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:26:37.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:26:37.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:26:37.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:26:37.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:26:37.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:26:37.830 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:26:37.852 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:37.851+0000 7f522ee1cb80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:26:37.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:26:38.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:26:38.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:26:38.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:26:38.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:26:38.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:26:38.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:26:38.129 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:26:38.129 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:26:38.130 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:26:38.238 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:26:38.239 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:26:38.340 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-01-10T07:26:38.340 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:38.340 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:38.340 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:38.340 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:38.341 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:38.341 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:38.341 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-01-10T07:26:38.341 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get osd_pool_default_size 2024-01-10T07:26:38.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:26:38.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:26:38.546 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:26:38.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:26:38.547 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:26:38.566 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:38.566+0000 7f742e2f5b80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:26:38.569 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:26:38.832 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:26:38.833 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:26:38.833 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:26:38.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:26:38.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:26:38.834 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:26:38.834 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:26:38.834 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:38.834 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:38.835 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:38.835 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:38.835 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:38.835 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:38.835 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:26:38.835 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:38.937 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:38.938 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:38.938 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:38.938 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:38.938 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:26:38.938 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:39.039 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:39.040 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-01-10T07:26:39.040 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get osd_pool_default_size 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:26:39.139 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:26:39.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:26:39.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-01-10T07:26:39.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:26:39.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:26:39.247 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:26:39.248 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:26:39.248 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:26:39.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:26:39.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:26:39.248 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:26:39.250 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:26:39.250 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:26:39.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:26:39.251 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:26:39.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:26:39.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:26:39.253 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:26:39.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:26:39.254 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:26:39.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:26:39.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:26:39.256 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:26:39.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:26:39.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_run_osd td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:26:39.269 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:26:39.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:26:39.271 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:26:39.271 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:26:39.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:26:39.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:26:39.272 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:26:39.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:26:39.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:26:39.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:26:39.274 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:26:39.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:26:39.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:26:39.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:26:39.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:26:39.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:39.277 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:39.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:26:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:26:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:26:39.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:26:39.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:26:39.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:39.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:39.279 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:26:39.281 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:26:39.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:26:39.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:26:39.282 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-01-10T07:26:39.644 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:26:39.645 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-01-10T07:26:39.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:26:39.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:26:39.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:26:39.955 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:26:39.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:26:39.956 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:26:39.956 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:26:39.956 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:39.956 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:39.957 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:39.957 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:39.957 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:39.957 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:39.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:26:39.958 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:26:40.062 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:26:40.062 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:26:40.062 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:26:40.062 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:26:40.062 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:26:40.063 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:26:40.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:26:41.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:26:41.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:26:41.059 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:26:41.060 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:26:41.060 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:41.060 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:41.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:26:41.060 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:26:41.060 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:26:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-01-10T07:26:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:26:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:26:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:26:41.088 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:26:41.089 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:26:41.090 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:26:41.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:26:41.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:26:41.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:26:41.091 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:26:41.091 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b0bebb5-0ccb-49ec-8e2c-9ac557aca0e9 2024-01-10T07:26:41.091 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 9b0bebb5-0ccb-49ec-8e2c-9ac557aca0e9 2024-01-10T07:26:41.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9b0bebb5-0ccb-49ec-8e2c-9ac557aca0e9' 2024-01-10T07:26:41.092 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:26:41.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxRp5lI6ZBBhAAgfeYkUjoufQeremP7h5Mgw== 2024-01-10T07:26:41.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxRp5lI6ZBBhAAgfeYkUjoufQeremP7h5Mgw=="}' 2024-01-10T07:26:41.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b0bebb5-0ccb-49ec-8e2c-9ac557aca0e9 -i td/ceph-helpers/0/new.json 2024-01-10T07:26:41.499 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:26:41.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:26:41.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxRp5lI6ZBBhAAgfeYkUjoufQeremP7h5Mgw== --osd-uuid 9b0bebb5-0ccb-49ec-8e2c-9ac557aca0e9 2024-01-10T07:26:41.536 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.536+0000 7f1ea5f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:41.538 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.538+0000 7f1ea5f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:41.539 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.538+0000 7f1ea5f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:41.540 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.539+0000 7f1ea5f12540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:26:41.540 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.539+0000 7f1ea5f12540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:26:41.540 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.539+0000 7f1ea5f12540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:26:41.613 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:41.612+0000 7f1ea5f12540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:26:46.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:26:46.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:26:46.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:26:46.887 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:26:46.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:26:47.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:26:47.164 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:26:47.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:26:47.164 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:26:47.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:26:47.165 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:26:47.186 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:47.186+0000 7f4d39485540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:47.245 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:47.244+0000 7f4d39485540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:47.245 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:47.245+0000 7f4d39485540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:26:47.435 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:26:47.436 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:26:47.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:47.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:47.784 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:47.784+0000 7f4d39485540 -1 Falling back to public interface 2024-01-10T07:26:47.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:48.907 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:48.908 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:48.908 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:26:48.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:26:48.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:48.909 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:49.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:50.176 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:26:50.176+0000 7f4d39485540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:26:50.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:50.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:50.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:26:50.381 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:26:50.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:50.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:50.898 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:51.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:51.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:51.899 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:26:51.900 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:26:51.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:51.900 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:52.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:53.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:53.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:53.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:26:53.367 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:26:53.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:53.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:53.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:54.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:54.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:54.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:26:54.894 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:26:54.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:54.894 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:55.350 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:56.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:56.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:56.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:26:56.353 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:26:56.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:56.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:56.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:57.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:57.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:57.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:26:57.861 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:26:57.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:57.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:58.323 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:26:59.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:26:59.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:26:59.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:26:59.325 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:26:59.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:26:59.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:26:59.859 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:00.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:00.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:00.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:27:00.861 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:27:00.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:00.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:01.488 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:02.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:02.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:02.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:27:02.490 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:27:02.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:02.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:03.417 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:04.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:04.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:04.419 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:27:04.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:27:04.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:04.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:05.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:06.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:06.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:06.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:27:06.123 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:27:06.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:06.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:06.856 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:07.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:07.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:07.857 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:27:07.858 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:27:07.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:07.858 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:08.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:09.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:09.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:09.499 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:27:09.499 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:27:09.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:09.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:10.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:11.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:11.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:11.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:27:11.298 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:27:11.298 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:11.299 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:11.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:12.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:12.826 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:12.826 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:27:12.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:27:12.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:12.827 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:27:13.303 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/155604,v1:127.0.0.1:6803/155604] [v2:127.0.0.1:6804/155604,v1:127.0.0.1:6805/155604] exists,up 9b0bebb5-0ccb-49ec-8e2c-9ac557aca0e9 2024-01-10T07:27:13.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:27:13.305 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.0.asok 2024-01-10T07:27:13.306 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-01-10T07:27:13.306 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.0.asok config get osd_max_backfills 2024-01-10T07:27:13.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-01-10T07:27:13.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-01-10T07:27:13.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-01-10T07:27:13.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-01-10T07:27:13.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:27:13.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:27:13.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:27:13.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:27:13.419 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:27:13.420 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-01-10T07:27:13.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:27:13.422 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:27:13.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=001003e2-cb1d-4c3d-9e55-4ecda9dca5a6 2024-01-10T07:27:13.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 001003e2-cb1d-4c3d-9e55-4ecda9dca5a6' 2024-01-10T07:27:13.422 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 001003e2-cb1d-4c3d-9e55-4ecda9dca5a6 2024-01-10T07:27:13.422 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:27:13.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRRp5lcNrCGRAALng8REFnjtp9WlvOrcidyA== 2024-01-10T07:27:13.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRRp5lcNrCGRAALng8REFnjtp9WlvOrcidyA=="}' 2024-01-10T07:27:13.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 001003e2-cb1d-4c3d-9e55-4ecda9dca5a6 -i td/ceph-helpers/1/new.json 2024-01-10T07:27:13.961 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:27:13.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:27:13.974 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQDRRp5lcNrCGRAALng8REFnjtp9WlvOrcidyA== --osd-uuid 001003e2-cb1d-4c3d-9e55-4ecda9dca5a6 2024-01-10T07:27:14.001 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:13.999+0000 7ffb0176e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:14.001 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:14.001+0000 7ffb0176e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:14.002 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:14.002+0000 7ffb0176e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:14.002 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:14.002+0000 7ffb0176e540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:27:14.002 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:14.002+0000 7ffb0176e540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:27:14.002 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:14.002+0000 7ffb0176e540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:27:14.083 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:14.083+0000 7ffb0176e540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:27:19.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:27:19.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:27:19.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:27:19.926 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:27:19.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:27:20.552 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:27:20.553 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:27:20.553 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-01-10T07:27:20.553 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:27:20.553 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:27:20.554 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:27:20.576 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:20.576+0000 7fdcdff7b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:20.579 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:20.578+0000 7fdcdff7b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:20.580 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:20.579+0000 7fdcdff7b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:21.074 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:27:21.075 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:27:21.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:21.075 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:21.107 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:21.107+0000 7fdcdff7b540 -1 Falling back to public interface 2024-01-10T07:27:21.599 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:22.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:22.604 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:22.605 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:27:22.605 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:27:22.606 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:22.606 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:23.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:23.708 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:23.707+0000 7fdcdff7b540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:27:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:24.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:24.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:27:24.129 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:27:24.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:24.129 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:24.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:25.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:25.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:25.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:27:25.744 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:27:25.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:25.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:26.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:27.212 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:27:27.212 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:27:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:27.213 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:27.860 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:28.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:28.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:28.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:27:28.861 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:27:28.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:28.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:29.327 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:30.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:30.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:30.328 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:27:30.329 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:27:30.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:30.329 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:30.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:31.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:31.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:31.933 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:27:31.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:27:31.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:31.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:32.411 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:33.412 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:33.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:33.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:27:33.413 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:27:33.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:33.413 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:33.920 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:34.921 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:34.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:34.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:27:34.922 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:27:34.922 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:34.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:35.940 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:36.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:36.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:36.942 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:27:36.942 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:27:36.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:36.943 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:37.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:38.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:38.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:38.854 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:27:38.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:27:38.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:38.854 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:39.708 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:40.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:40.709 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:40.710 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:27:40.710 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:27:40.710 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:40.710 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:41.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:42.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:42.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:42.792 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:27:42.793 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:27:42.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:42.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:43.534 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:44.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:44.535 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:44.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:27:44.536 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:27:44.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:44.536 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:45.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:46.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:46.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:46.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:27:46.225 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:27:46.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:46.225 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:27:46.696 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/157424,v1:127.0.0.1:6811/157424] [v2:127.0.0.1:6812/157424,v1:127.0.0.1:6813/157424] exists,up 001003e2-cb1d-4c3d-9e55-4ecda9dca5a6 2024-01-10T07:27:46.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:27:46.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:27:46.697 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:27:46.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-01-10T07:27:46.697 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.1.asok 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-01-10T07:27:46.698 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.1.asok config get osd_max_backfills 2024-01-10T07:27:46.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:27:46.809 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:27:46.810 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:27:46.811 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-01-10T07:27:46.822 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:27:46.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cb07e24-f768-42d0-9340-93b64412d1f9 2024-01-10T07:27:46.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5cb07e24-f768-42d0-9340-93b64412d1f9' 2024-01-10T07:27:46.823 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 5cb07e24-f768-42d0-9340-93b64412d1f9 2024-01-10T07:27:46.823 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:27:46.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyRp5l+ckSMhAAgewONyI8FyNDlkvfx2NAZw== 2024-01-10T07:27:46.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyRp5l+ckSMhAAgewONyI8FyNDlkvfx2NAZw=="}' 2024-01-10T07:27:46.840 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cb07e24-f768-42d0-9340-93b64412d1f9 -i td/ceph-helpers/2/new.json 2024-01-10T07:27:47.427 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:27:47.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-01-10T07:27:47.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyRp5l+ckSMhAAgewONyI8FyNDlkvfx2NAZw== --osd-uuid 5cb07e24-f768-42d0-9340-93b64412d1f9 2024-01-10T07:27:47.468 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.467+0000 7f98ea8af540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:47.486 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.486+0000 7f98ea8af540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:47.487 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.487+0000 7f98ea8af540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:47.487 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.487+0000 7f98ea8af540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:27:47.487 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.487+0000 7f98ea8af540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:27:47.487 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.487+0000 7f98ea8af540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:27:47.570 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:47.570+0000 7f98ea8af540 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-01-10T07:27:53.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-01-10T07:27:53.626 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:27:53.627 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T07:27:53.627 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T07:27:53.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:27:54.290 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T07:27:54.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T07:27:54.290 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:27:54.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:27:54.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:27:54.291 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:27:54.315 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:54.315+0000 7f94aa8c1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:54.338 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:54.338+0000 7f94aa8c1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:54.339 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:54.339+0000 7f94aa8c1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:54.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:27:54.778 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:27:54.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:54.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:27:54.862 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:54.862+0000 7f94aa8c1540 -1 Falling back to public interface 2024-01-10T07:27:55.285 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:56.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:56.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:56.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:27:56.287 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:27:56.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:56.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:27:56.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:57.335 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:27:57.334+0000 7f94aa8c1540 -1 osd.2 0 log_to_monitors true 2024-01-10T07:27:57.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:57.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:57.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:27:57.774 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:27:57.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:57.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:27:58.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:27:59.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:27:59.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:27:59.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:27:59.314 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:27:59.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:27:59.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:27:59.793 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:00.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:00.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:00.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:28:00.795 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:28:00.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:00.795 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:01.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:02.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:02.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:02.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:28:02.264 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:28:02.265 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:02.265 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:02.748 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:03.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:03.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:03.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:28:03.750 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:28:03.750 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:03.751 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:04.284 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:05.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:05.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:05.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:28:05.286 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:28:05.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:05.287 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:05.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:06.776 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:06.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:06.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:28:06.777 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:28:06.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:06.778 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:07.293 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:08.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:08.294 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:08.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:28:08.295 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:28:08.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:08.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:08.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:09.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:09.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:09.820 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:28:09.821 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:28:09.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:09.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:10.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:11.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:11.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:11.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:28:11.784 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:28:11.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:11.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:12.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:13.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:13.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:13.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:28:13.308 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:28:13.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:13.309 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:14.131 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:15.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:15.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:15.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:28:15.133 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:28:15.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:15.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:15.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:16.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:16.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:16.731 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:28:16.731 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:28:16.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:16.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:17.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:18.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:18.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:18.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:28:18.241 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:28:18.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:18.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:18.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:19.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:19.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:19.775 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:28:19.776 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:28:19.776 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:19.777 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:20.237 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:21.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:21.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:21.239 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:28:21.240 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:28:21.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:21.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:28:21.752 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/159159,v1:127.0.0.1:6819/159159] [v2:127.0.0.1:6820/159159,v1:127.0.0.1:6821/159159] exists,up 5cb07e24-f768-42d0-9340-93b64412d1f9 2024-01-10T07:28:21.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:28:21.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:28:21.754 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:28:21.754 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-osd.2.asok 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-01-10T07:28:21.755 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.87895/ceph-osd.2.asok config get osd_max_backfills 2024-01-10T07:28:21.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-01-10T07:28:21.861 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:21.862 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:21.863 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:21.968 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:21.969 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:21.969 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:21.969 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:21.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:21.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:21.971 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:21.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:21.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:21.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:21.973 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:21.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:21.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:21.975 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:21.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:21.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:21.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:21.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_set_config td/ceph-helpers 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-01-10T07:28:21.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:21.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:21.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:21.990 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:21.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:21.990 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:21.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:21.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:21.991 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:21.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:21.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:21.994 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:21.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:21.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:21.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:21.997 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:21.997 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:21.997 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:21.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:21.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:21.997 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:21.998 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:28:21.999 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:28:21.999 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:21.999 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:21.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:28:22.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:28:22.002 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:28:22.002 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:28:22.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-01-10T07:28:22.606 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:28:22.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:28:22.607 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:22.608 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:22.608 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:28:22.608 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:28:22.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:22.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:22.608 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:22.609 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:22.609 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:22.609 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:22.609 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:22.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:22.609 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:28:22.715 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:28:22.715 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:22.715 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:22.715 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:22.716 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:28:22.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-01-10T07:28:22.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:22.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:22.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-01-10T07:28:22.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:22.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get ms_crc_header 2024-01-10T07:28:22.918 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:22.919 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:22.920 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config set ms_crc_header false 2024-01-10T07:28:23.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-01-10T07:28:23.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-01-10T07:28:23.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:23.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:23.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-01-10T07:28:23.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.021 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:23.022 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:23.022 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get ms_crc_header 2024-01-10T07:28:23.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-01-10T07:28:23.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-01-10T07:28:23.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-01-10T07:28:23.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:23.123 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config set ms_crc_header true 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:23.224 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:23.225 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.225 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.225 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:23.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:23.225 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get ms_crc_header 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:23.324 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:23.325 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:23.428 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:23.428 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:23.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:23.429 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:23.430 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:23.430 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:23.431 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:23.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.432 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:23.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:23.433 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:23.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:23.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:23.436 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:23.436 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.436 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_setup td/ceph-helpers 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:28:23.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:23.439 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:23.440 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:23.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:23.440 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:23.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:23.442 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:23.442 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:23.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:23.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:23.444 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:23.445 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:23.446 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:23.447 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:23.447 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.447 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:23.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:23.447 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:23.448 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:28:23.449 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:28:23.449 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.449 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:28:23.451 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:28:23.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:23.452 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:23.453 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:23.453 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:23.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:23.454 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:23.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:23.455 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:23.455 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:23.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:23.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:23.457 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:23.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:23.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:23.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:23.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.460 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.460 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:23.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:23.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:23.461 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:28:23.462 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:28:23.462 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.462 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.462 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:28:23.464 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:23.465 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:23.466 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:23.466 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:23.467 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:23.467 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:23.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:23.468 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:23.468 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:23.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:23.470 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:23.471 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:23.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:23.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:23.473 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:23.473 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.473 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:23.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:23.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:23.475 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_teardown td/ceph-helpers 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:23.476 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:23.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:23.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:23.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:23.478 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:23.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:23.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:23.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:23.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:23.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:23.481 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.481 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:23.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:23.483 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:23.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:23.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.484 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:23.485 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:23.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:23.486 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:28:23.487 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:28:23.487 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.487 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:23.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:23.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:23.490 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:23.490 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:23.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:23.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:23.490 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:23.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:23.491 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:23.491 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:23.491 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:23.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:23.492 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:23.492 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:23.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:23.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:23.494 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:23.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:23.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:23.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:23.497 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:23.497 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:23.497 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:23.497 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_wait_background td/ceph-helpers 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: test_wait_background: local pids= 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2024-01-10T07:28:23.500 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 160352"' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 160352' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 1; exit 1' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 160354"' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 160354' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/160355: /' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_wait_background: wait_background pids 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: wait_background: pids=' 160352 160354' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_background: return_code=0 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 160352 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 2; exit 0' 2024-01-10T07:28:23.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/160357: /' 2024-01-10T07:28:24.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 1 2024-01-10T07:28:24.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2101: wait_background: return_code=1 2024-01-10T07:28:24.501 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2024-01-10T07:28:24.502 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 160354 2024-01-10T07:28:25.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2024-01-10T07:28:25.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: eval 'pids='\'''\''' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: pids= 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: wait_background: return 1 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: test_wait_background: '[' 1 -ne 1 ']' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2119: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 160360"' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 160360' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 1; exit 0' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 160362"' 2024-01-10T07:28:25.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 160362' 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: test_wait_background: wait_background pids 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/160363: /' 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: wait_background: pids=' 160360 160362' 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_background: return_code=0 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 160360 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 2; exit 0' 2024-01-10T07:28:25.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/160365: /' 2024-01-10T07:28:26.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2024-01-10T07:28:26.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2024-01-10T07:28:26.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 160362 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: eval 'pids='\'''\''' 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: pids= 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: wait_background: return 0 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: test_wait_background: '[' 0 -ne 0 ']' 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: test_wait_background: '[' '!' -z '' ']' 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_wait_for_clean td/ceph-helpers 2024-01-10T07:28:27.507 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-01-10T07:28:27.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-01-10T07:28:27.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:28:27.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:28:27.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:28:27.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:28:27.508 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:28:27.509 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:28:27.509 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:28:27.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:28:27.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:28:27.509 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:28:27.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:28:27.510 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:28:27.510 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:28:27.510 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:28:27.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:28:27.511 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:28:27.511 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:28:27.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:27.513 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:28:27.514 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:28:27.514 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:28:27.515 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:28:27.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:28:27.516 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:28:27.517 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:28:27.517 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:27.518 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:27.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:28:27.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:28:27.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:28:27.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:28:27.520 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:28:27.520 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:27.520 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:27.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:28:27.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:28:27.523 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:28:27.524 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:27.894 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:28:27.895 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-01-10T07:28:28.154 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:28:28.155 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:28:28.155 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:28.155 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:28.155 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:28:28.155 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:28:28.156 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:28.156 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:28.156 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:28.157 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:28.157 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:28.157 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:28.157 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:28.157 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:28.157 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:28:28.270 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:28:28.271 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:28:28.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:28:28.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:28:28.376 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9cb1f0f0-332a-469e-b5d4-3557e56c2940 2024-01-10T07:28:28.377 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9cb1f0f0-332a-469e-b5d4-3557e56c2940' 2024-01-10T07:28:28.378 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 9cb1f0f0-332a-469e-b5d4-3557e56c2940 2024-01-10T07:28:28.378 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:28:28.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcR55lOdc+FxAAymm6+3LbA8b7RzDI4irWcQ== 2024-01-10T07:28:28.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcR55lOdc+FxAAymm6+3LbA8b7RzDI4irWcQ=="}' 2024-01-10T07:28:28.390 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9cb1f0f0-332a-469e-b5d4-3557e56c2940 -i td/ceph-helpers/0/new.json 2024-01-10T07:28:29.353 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:28:29.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:28:29.365 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcR55lOdc+FxAAymm6+3LbA8b7RzDI4irWcQ== --osd-uuid 9cb1f0f0-332a-469e-b5d4-3557e56c2940 2024-01-10T07:28:29.389 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.389+0000 7f80170ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:28:29.391 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.391+0000 7f80170ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:28:29.392 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.392+0000 7f80170ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:28:29.392 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.392+0000 7f80170ad540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:28:29.392 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.392+0000 7f80170ad540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:28:29.392 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.392+0000 7f80170ad540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:28:29.469 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:29.469+0000 7f80170ad540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:28:34.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:28:34.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:28:34.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:28:34.773 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:28:34.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:28:35.047 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:28:35.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:28:35.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:28:35.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:28:35.048 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:28:35.049 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:28:35.070 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:35.070+0000 7fec5fd2d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:28:35.105 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:35.104+0000 7fec5fd2d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:28:35.105 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:35.105+0000 7fec5fd2d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:35.315 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:28:35.316 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:28:35.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:35.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:35.542 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:35.623 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:35.623+0000 7fec5fd2d540 -1 Falling back to public interface 2024-01-10T07:28:36.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:36.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:36.543 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:28:36.543 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:28:36.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:36.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:36.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:37.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:37.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:37.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:28:37.788 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:28:37.788 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:37.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:37.997 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:28:37.997+0000 7fec5fd2d540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:28:38.030 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:39.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:39.031 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:39.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:28:39.032 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:28:39.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:39.032 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:39.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:40.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:40.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:40.275 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:28:40.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:28:40.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:40.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:40.518 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:41.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:41.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:41.519 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:28:41.519 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:28:41.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:41.520 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:41.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:42.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:42.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:42.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:28:42.765 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:28:42.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:42.765 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:43.007 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:44.008 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:28:44.009 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:28:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:44.009 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:44.250 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:45.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:45.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:45.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:28:45.252 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:28:45.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:45.252 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:45.494 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:46.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:46.495 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:46.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:28:46.496 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:28:46.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:46.496 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:46.739 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:47.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:47.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:47.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:28:47.741 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:28:47.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:47.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:47.983 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:48.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:48.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:48.984 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:28:48.985 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:28:48.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:48.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:49.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:50.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:50.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:50.229 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:28:50.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:28:50.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:50.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:50.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:51.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:51.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:51.473 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:28:51.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:28:51.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:51.474 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:51.715 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:52.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:52.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:52.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:28:52.717 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:28:52.717 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:52.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:52.962 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:53.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:53.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:53.964 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:28:53.964 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:28:53.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:53.965 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:54.205 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:55.206 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:55.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:55.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:28:55.207 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:28:55.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:55.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:55.449 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:56.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:56.450 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:56.450 INFO:tasks.workunit.client.0.smithi057.stdout:17 2024-01-10T07:28:56.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-01-10T07:28:56.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:56.451 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:56.692 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:57.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:57.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:57.694 INFO:tasks.workunit.client.0.smithi057.stdout:18 2024-01-10T07:28:57.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-01-10T07:28:57.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:57.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:57.947 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:28:58.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:28:58.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:28:58.951 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-01-10T07:28:58.951 INFO:tasks.workunit.client.0.smithi057.stdout:19 2024-01-10T07:28:58.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:28:58.952 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:28:59.190 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/161307,v1:127.0.0.1:6801/161307] [v2:127.0.0.1:6802/161307,v1:127.0.0.1:6803/161307] exists,up 9cb1f0f0-332a-469e-b5d4-3557e56c2940 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:28:59.191 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:28:59.479 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:28:59.480 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:28:59.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:28:59.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-01-10T07:28:59.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-01-10T07:28:59.779 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' does not exist 2024-01-10T07:28:59.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-01-10T07:28:59.789 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-01-10T07:29:00.272 INFO:tasks.workunit.client.0.smithi057.stderr:pool 'rbd' already exists 2024-01-10T07:29:00.281 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-01-10T07:29:01.283 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-01-10T07:29:02.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-01-10T07:29:02.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-01-10T07:29:02.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:29:02.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:29:02.100 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:29:02.101 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:29:02.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-01-10T07:29:02.101 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:29:02.101 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:29:02.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:29:02.101 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:29:02.102 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:29:02.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:29:02.127 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:29:02.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:29:02.128 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:29:02.128 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:29:02.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2024-01-10T07:29:02.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:29:02.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:29:02.362 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:29:02.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=0 2024-01-10T07:29:02.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 0 2024-01-10T07:29:02.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-0' 2024-01-10T07:29:02.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:29:02.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-0 2024-01-10T07:29:02.503 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:29:02.504 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:29:02.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-0 2024-01-10T07:29:02.504 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:29:02.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=0 2024-01-10T07:29:02.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-01-10T07:29:02.506 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 0 2024-01-10T07:29:02.506 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:29:02.740 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 0 -lt 0 2024-01-10T07:29:02.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:29:02.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:02.740 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:03.018 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-01-10T07:29:03.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-01-10T07:29:04.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:29:04.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:04.020 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:04.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-01-10T07:29:04.277 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-01-10T07:29:05.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:29:05.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:05.279 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:05.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-01-10T07:29:05.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-01-10T07:29:06.547 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:29:06.547 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:06.547 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:07.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:29:07.186 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:29:07.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:29:07.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:29:07.186 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:29:07.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:29:07.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:29:07.187 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:29:07.652 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-01-10T07:29:07.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:29:07.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:07.653 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:08.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-01-10T07:29:08.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-01-10T07:29:08.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-01-10T07:29:08.274 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-01-10T07:29:08.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:29:08.275 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-01-10T07:29:08.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:29:08.375 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:29:08.375 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:29:08.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:29:08.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:29:08.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:29:08.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:29:08.376 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:29:08.813 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-01-10T07:29:08.813 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:29:08.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:08.814 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-01-10T07:29:09.425 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-01-10T07:29:10.056 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-01-10T07:29:10.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-01-10T07:29:10.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-01-10T07:29:10.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:29:10.057 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-01-10T07:29:10.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:29:10.257 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:29:10.257 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:29:10.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:29:10.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:29:10.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:29:10.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:29:10.258 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:29:10.701 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-01-10T07:29:10.701 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:29:10.701 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:10.701 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:11.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-01-10T07:29:11.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-01-10T07:29:11.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-01-10T07:29:11.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-01-10T07:29:11.313 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-01-10T07:29:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-01-10T07:29:11.314 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-01-10T07:29:11.314 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-01-10T07:29:11.314 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-01-10T07:29:11.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-01-10T07:29:11.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-01-10T07:29:11.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-01-10T07:29:11.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:29:11.973 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-01-10T07:29:12.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:29:12.374 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:29:12.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:29:12.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:29:12.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:29:12.374 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:29:12.375 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:29:12.375 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:29:12.812 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-01-10T07:29:12.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:29:12.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:12.812 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-01-10T07:29:13.434 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-01-10T07:29:14.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-01-10T07:29:14.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-01-10T07:29:14.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-01-10T07:29:14.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-01-10T07:29:14.093 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:29:14.394 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:29:14.395 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:29:14.841 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-01-10T07:29:14.841 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:29:14.841 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:14.841 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-01-10T07:29:15.454 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-01-10T07:29:16.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-01-10T07:29:16.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-01-10T07:29:16.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-01-10T07:29:16.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-01-10T07:29:16.749 INFO:tasks.workunit.client.0.smithi057.stdout:{ 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "cluster_fingerprint": "ed81182c-1d96-4f23-9932-69e3140175b7", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "version": "17.2.7-268-g6441305d", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "commit": "6441305d08929b8a72c0322a5c268c64b4a99c65", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "timestamp": "2024-01-10T07:29:16.749166+0000", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "tag": "", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "health": { 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "status": "HEALTH_ERR", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "checks": { 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "MGR_MODULE_ERROR": { 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "severity": "HEALTH_ERR", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "summary": { 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "message": "2 mgr modules have failed", 2024-01-10T07:29:16.750 INFO:tasks.workunit.client.0.smithi057.stdout: "count": 2 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: "detail": [ 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: "message": "Module 'rbd_support' has failed: Not found or unloadable" 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: "message": "Module 'volumes' has failed: Not found or unloadable" 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: "muted": false 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: "PG_DEGRADED": { 2024-01-10T07:29:16.751 INFO:tasks.workunit.client.0.smithi057.stdout: "severity": "HEALTH_WARN", 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "summary": { 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "message": "Degraded data redundancy: 1/2 objects degraded (50.000%), 1 pg degraded", 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "count": 1 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "detail": [ 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "muted": false 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "TOO_FEW_OSDS": { 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "severity": "HEALTH_WARN", 2024-01-10T07:29:16.752 INFO:tasks.workunit.client.0.smithi057.stdout: "summary": { 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "count": 2 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "detail": [], 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "muted": false 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "mutes": [] 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "monmap_first_committed": 1, 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "monmap_last_committed": 1, 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "monmap": { 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "epoch": 1, 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "fsid": "c182505b-1213-4a65-a0b1-c14e269ca5fa", 2024-01-10T07:29:16.753 INFO:tasks.workunit.client.0.smithi057.stdout: "modified": "2024-01-10T07:28:27.545190Z", 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "created": "2024-01-10T07:28:27.545190Z", 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "min_mon_release": 17, 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "min_mon_release_name": "quincy", 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "election_strategy": 1, 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "disallowed_leaders: ": "", 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "stretch_mode": false, 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "tiebreaker_mon": "", 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "removed_ranks: ": "", 2024-01-10T07:29:16.754 INFO:tasks.workunit.client.0.smithi057.stdout: "features": { 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "persistent": [ 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "kraken", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "luminous", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "mimic", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "osdmap-prune", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "nautilus", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "octopus", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "pacific", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "elector-pinging", 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "quincy" 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: "optional": [] 2024-01-10T07:29:16.755 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "mons": [ 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "rank": 0, 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "a", 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "public_addrs": { 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "addrvec": [ 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v2", 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:7109", 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 0 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.756 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:7109/0", 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "public_addr": "127.0.0.1:7109/0", 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "priority": 0, 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 0, 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "crush_location": "{}" 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "quorum": [ 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: 0 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "osdmap": { 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "epoch": 13, 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "fsid": "c182505b-1213-4a65-a0b1-c14e269ca5fa", 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "created": "2024-01-10T07:28:28.425092+0000", 2024-01-10T07:29:16.757 INFO:tasks.workunit.client.0.smithi057.stdout: "modified": "2024-01-10T07:29:01.857804+0000", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "last_up_change": "2024-01-10T07:28:58.052766+0000", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "last_in_change": "2024-01-10T07:28:29.181279+0000", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "flags_num": 5799936, 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "flags_set": [ 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "pglog_hardlimit", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "purged_snapdirs", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "recovery_deletes", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "sortbitwise" 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "crush_version": 3, 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "full_ratio": 0.99000000953674316, 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "nearfull_ratio": 0.85000002384185791, 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "cluster_snapshot": "", 2024-01-10T07:29:16.758 INFO:tasks.workunit.client.0.smithi057.stdout: "pool_max": 1, 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "max_osd": 1, 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "require_min_compat_client": "luminous", 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "min_compat_client": "jewel", 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "require_osd_release": "quincy", 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "pools": [ 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "pool": 1, 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "pool_name": "rbd", 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "create_time": "2024-01-10T07:28:59.999689+0000", 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "flags": 8193, 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "type": 1, 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "size": 2, 2024-01-10T07:29:16.759 INFO:tasks.workunit.client.0.smithi057.stdout: "min_size": 1, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "crush_rule": 0, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "peering_crush_bucket_count": 0, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "peering_crush_bucket_target": 0, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "peering_crush_bucket_barrier": 0, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "object_hash": 2, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_autoscale_mode": "off", 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_num": 4, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_placement_num": 4, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_placement_num_target": 4, 2024-01-10T07:29:16.760 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_num_target": 4, 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_num_pending": 4, 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_pg_merge_meta": { 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "source_pgid": "0.0", 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "ready_epoch": 0, 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_epoch_started": 0, 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_epoch_clean": 0, 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "source_version": "0'0", 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "target_version": "0'0" 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_change": "13", 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_force_op_resend": "0", 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_force_op_resend_prenautilus": "0", 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "last_force_op_resend_preluminous": "0", 2024-01-10T07:29:16.761 INFO:tasks.workunit.client.0.smithi057.stdout: "auid": 0, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "snap_mode": "selfmanaged", 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "snap_seq": 3, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "snap_epoch": 13, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "pool_snaps": [], 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "removed_snaps": "[]", 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "quota_max_bytes": 0, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "quota_max_objects": 0, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "tiers": [], 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "tier_of": -1, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "read_tier": -1, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "write_tier": -1, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "cache_mode": "none", 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "target_max_bytes": 0, 2024-01-10T07:29:16.762 INFO:tasks.workunit.client.0.smithi057.stdout: "target_max_objects": 0, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "cache_target_full_ratio_micro": 800000, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "cache_min_flush_age": 0, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "cache_min_evict_age": 0, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "erasure_code_profile": "", 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "hit_set_params": { 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "none" 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "hit_set_period": 0, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "hit_set_count": 0, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "use_gmt_hitset": true, 2024-01-10T07:29:16.763 INFO:tasks.workunit.client.0.smithi057.stdout: "min_read_recency_for_promote": 0, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "min_write_recency_for_promote": 0, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "hit_set_grade_decay_rate": 0, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "hit_set_search_last_n": 0, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "grade_table": [], 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "stripe_width": 0, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "expected_num_objects": 0, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "fast_read": false, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "options": {}, 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "application_metadata": { 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: "rbd": {} 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.764 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "osds": [ 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "osd": 0, 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "uuid": "9cb1f0f0-332a-469e-b5d4-3557e56c2940", 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "up": 1, 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "in": 1, 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 1, 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "primary_affinity": 1, 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "last_clean_begin": 0, 2024-01-10T07:29:16.765 INFO:tasks.workunit.client.0.smithi057.stdout: "last_clean_end": 0, 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "up_from": 5, 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "up_thru": 7, 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "down_at": 0, 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "lost_at": 0, 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "public_addrs": { 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "addrvec": [ 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v2", 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6800", 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.766 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v1", 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6801", 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "cluster_addrs": { 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "addrvec": [ 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v2", 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6802", 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v1", 2024-01-10T07:29:16.767 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6803", 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: "heartbeat_back_addrs": { 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: "addrvec": [ 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v2", 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6806", 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.768 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.769 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v1", 2024-01-10T07:29:16.769 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6807", 2024-01-10T07:29:16.769 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.769 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "heartbeat_front_addrs": { 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "addrvec": [ 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v2", 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6804", 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "v1", 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "addr": "127.0.0.1:6805", 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: "nonce": 161307 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.770 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "public_addr": "127.0.0.1:6801/161307", 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "cluster_addr": "127.0.0.1:6803/161307", 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "heartbeat_back_addr": "127.0.0.1:6807/161307", 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "heartbeat_front_addr": "127.0.0.1:6805/161307", 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "state": [ 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "exists", 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "up" 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_xinfo": [ 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "osd": 0, 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "down_stamp": "0.000000", 2024-01-10T07:29:16.771 INFO:tasks.workunit.client.0.smithi057.stdout: "laggy_probability": 0, 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "laggy_interval": 0, 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "features": 4540138320759226367, 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "old_weight": 0, 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "last_purged_snaps_scrub": "2024-01-10T07:28:57.788065+0000", 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "dead_epoch": 0 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_upmap": [], 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_upmap_items": [], 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "pg_temp": [], 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "primary_temp": [], 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "blocklist": {}, 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "range_blocklist": {}, 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "erasure_code_profiles": { 2024-01-10T07:29:16.772 INFO:tasks.workunit.client.0.smithi057.stdout: "default": { 2024-01-10T07:29:16.773 INFO:tasks.workunit.client.0.smithi057.stdout: "k": "2", 2024-01-10T07:29:16.773 INFO:tasks.workunit.client.0.smithi057.stdout: "m": "2", 2024-01-10T07:29:16.773 INFO:tasks.workunit.client.0.smithi057.stdout: "plugin": "jerasure", 2024-01-10T07:29:16.773 INFO:tasks.workunit.client.0.smithi057.stdout: "technique": "reed_sol_van" 2024-01-10T07:29:16.773 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.773 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "removed_snaps_queue": [ 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "pool": 1, 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "snaps": [ 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "begin": 2, 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "length": 1 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "new_removed_snaps": [ 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "pool": 1, 2024-01-10T07:29:16.774 INFO:tasks.workunit.client.0.smithi057.stdout: "snaps": [ 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "begin": 2, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "length": 1 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "new_purged_snaps": [], 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "crush_node_flags": {}, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "device_class_flags": {}, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "stretch_mode": { 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "stretch_mode_enabled": false, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "stretch_bucket_count": 0, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "degraded_stretch_mode": 0, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "recovering_stretch_mode": 0, 2024-01-10T07:29:16.775 INFO:tasks.workunit.client.0.smithi057.stdout: "stretch_mode_bucket": 0 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_metadata": [ 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "id": 0, 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "arch": "x86_64", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "back_addr": "[v2:127.0.0.1:6802/161307,v1:127.0.0.1:6803/161307]", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "back_iface": "", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "bluefs": "1", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "bluefs_dedicated_db": "0", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "bluefs_dedicated_wal": "0", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "bluefs_single_shared_device": "1", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_access_mode": "file", 2024-01-10T07:29:16.776 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_block_size": "4096", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_devices": "sda", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_rotational": "1", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_size": "107374182400", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_support_discard": "0", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_bdev_type": "hdd", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "bluestore_min_alloc_size": "4096", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "ceph_release": "quincy", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "ceph_version": "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "ceph_version_short": "17.2.7-268-g6441305d", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "ceph_version_when_created": "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "created_at": "2024-01-10T07:28:34.118996Z", 2024-01-10T07:29:16.777 INFO:tasks.workunit.client.0.smithi057.stdout: "default_device_class": "hdd", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4HQFM", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "devices": "sda", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "distro": "centos", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "distro_description": "CentOS Stream 8", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "distro_version": "8", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "front_addr": "[v2:127.0.0.1:6800/161307,v1:127.0.0.1:6801/161307]", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "front_iface": "", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/161307,v1:127.0.0.1:6807/161307]", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/161307,v1:127.0.0.1:6805/161307]", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "hostname": "smithi057", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "journal_rotational": "1", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "kernel_description": "#1 SMP Thu Dec 21 12:11:59 UTC 2023", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "kernel_version": "4.18.0-532.el8.x86_64", 2024-01-10T07:29:16.778 INFO:tasks.workunit.client.0.smithi057.stdout: "mem_swap_kb": "0", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "mem_total_kb": "32543872", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "objectstore_numa_unknown_devices": "sda", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "os": "Linux", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_data": "td/ceph-helpers/0", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_objectstore": "bluestore", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "osdspec_affinity": "", 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "rotational": "1" 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "osdmap_clean_epochs": { 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "min_last_epoch_clean": 0, 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "last_epoch_clean": { 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: "per_pool": [] 2024-01-10T07:29:16.779 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_epochs": [ 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "id": 0, 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "epoch": 5 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "osdmap_first_committed": 1, 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "osdmap_last_committed": 13, 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "crushmap": { 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "devices": [ 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "id": 0, 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "osd.0", 2024-01-10T07:29:16.780 INFO:tasks.workunit.client.0.smithi057.stdout: "class": "hdd" 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "types": [ 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 0, 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "osd" 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 1, 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "host" 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 2, 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "chassis" 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.781 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 3, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "rack" 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 4, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "row" 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 5, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "pdu" 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 6, 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "pod" 2024-01-10T07:29:16.782 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 7, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "room" 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 8, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "datacenter" 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 9, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "zone" 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 10, 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "region" 2024-01-10T07:29:16.783 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 11, 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "root" 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "buckets": [ 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "id": -1, 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "default", 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 11, 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "type_name": "root", 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "alg": "straw2", 2024-01-10T07:29:16.784 INFO:tasks.workunit.client.0.smithi057.stdout: "hash": "rjenkins1", 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "items": [ 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "id": -3, 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "pos": 0 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "id": -2, 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "default~hdd", 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 11, 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "type_name": "root", 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "alg": "straw2", 2024-01-10T07:29:16.785 INFO:tasks.workunit.client.0.smithi057.stdout: "hash": "rjenkins1", 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "items": [ 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "id": -4, 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "pos": 0 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "id": -3, 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "smithi057", 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 1, 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "type_name": "host", 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.786 INFO:tasks.workunit.client.0.smithi057.stdout: "alg": "straw2", 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "hash": "rjenkins1", 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "items": [ 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "id": 0, 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "pos": 0 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "id": -4, 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "name": "smithi057~hdd", 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "type_id": 1, 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "type_name": "host", 2024-01-10T07:29:16.787 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "alg": "straw2", 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "hash": "rjenkins1", 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "items": [ 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "id": 0, 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "weight": 6402, 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "pos": 0 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "rules": [ 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.788 INFO:tasks.workunit.client.0.smithi057.stdout: "rule_id": 0, 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "rule_name": "replicated_rule", 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "type": 1, 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "steps": [ 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "op": "take", 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "item": -1, 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "item_name": "default" 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "op": "choose_firstn", 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "num": 0, 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "type": "osd" 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.789 INFO:tasks.workunit.client.0.smithi057.stdout: "op": "emit" 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: ] 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "tunables": { 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "choose_local_tries": 0, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "choose_local_fallback_tries": 0, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "choose_total_tries": 50, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "chooseleaf_descend_once": 1, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "chooseleaf_vary_r": 1, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "chooseleaf_stable": 1, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "straw_calc_version": 1, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "allowed_bucket_algs": 54, 2024-01-10T07:29:16.790 INFO:tasks.workunit.client.0.smithi057.stdout: "profile": "jewel", 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "optimal_tunables": 1, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "legacy_tunables": 0, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "minimum_required_version": "jewel", 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "require_feature_tunables": 1, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "require_feature_tunables2": 1, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "has_v2_rules": 0, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "require_feature_tunables3": 1, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "has_v3_rules": 0, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "has_v4_buckets": 1, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "require_feature_tunables5": 1, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "has_v5_rules": 0 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "choose_args": {} 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.791 INFO:tasks.workunit.client.0.smithi057.stdout: "fsmap": { 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "epoch": 1, 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "default_fscid": -1, 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "compat": { 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "compat": {}, 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "ro_compat": {}, 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "incompat": { 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_1": "base v0.20", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_2": "client writeable ranges", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_3": "default file layouts on dirs", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_4": "dir inode in separate object", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_5": "mds uses versioned encoding", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_6": "dirfrag is stored in omap", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_8": "no anchor table", 2024-01-10T07:29:16.792 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_9": "file layout v2", 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_10": "snaprealm v2" 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "feature_flags": { 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "enable_multiple": true, 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "ever_enabled_multiple": true 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "standbys": [], 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "filesystems": [] 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "mdsmap_first_committed": 1, 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "mdsmap_last_committed": 1, 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "auth": { 2024-01-10T07:29:16.793 INFO:tasks.workunit.client.0.smithi057.stdout: "first_committed": 1, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "last_committed": 3, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_secrets": 1 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pg": 4, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pg_active": 4, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pg_unknown": 0, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_osd": 1, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "pool_sum": { 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "stat_sum": { 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_bytes": 19, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects": 1, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_object_clones": 0, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_object_copies": 2, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_missing_on_primary": 0, 2024-01-10T07:29:16.794 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_missing": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_degraded": 1, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_misplaced": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_unfound": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_dirty": 1, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_whiteouts": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_read": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_read_kb": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_write": 2, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_write_kb": 2, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_scrub_errors": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_shallow_scrub_errors": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_deep_scrub_errors": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_recovered": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_bytes_recovered": 0, 2024-01-10T07:29:16.795 INFO:tasks.workunit.client.0.smithi057.stdout: "num_keys_recovered": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_omap": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_hit_set_archive": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_bytes_hit_set_archive": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush_kb": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict_kb": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_promote": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush_mode_high": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush_mode_low": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict_mode_some": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict_mode_full": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_pinned": 0, 2024-01-10T07:29:16.796 INFO:tasks.workunit.client.0.smithi057.stdout: "num_legacy_snapsets": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "num_large_omap_objects": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_manifest": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "num_omap_bytes": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "num_omap_keys": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_repaired": 0 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "store_stats": { 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "total": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "available": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "internally_reserved": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "allocated": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "data_stored": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_allocated": 0, 2024-01-10T07:29:16.797 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_original": 0, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "omap_allocated": 0, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "internal_metadata": 0 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "log_size": 2, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "ondisk_log_size": 2, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "up": 4, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "acting": 4, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "num_store_stats": 0 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_sum": { 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "up_from": 0, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "seq": 0, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pgs": 4, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "num_osds": 1, 2024-01-10T07:29:16.798 INFO:tasks.workunit.client.0.smithi057.stdout: "num_per_pool_osds": 1, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "num_per_pool_omap_osds": 1, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "kb": 104857600, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used": 297212, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used_data": 116, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used_omap": 0, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used_meta": 297088, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_avail": 104560388, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "statfs": { 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "total": 107374182400, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "available": 107069837312, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "internally_reserved": 0, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "allocated": 118784, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "data_stored": 24598, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed": 0, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_allocated": 0, 2024-01-10T07:29:16.799 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_original": 0, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "omap_allocated": 0, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "internal_metadata": 304218112 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "hb_peers": [], 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "snap_trim_queue_len": 0, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "num_snap_trimming": 0, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "num_shards_repaired": 0, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "op_queue_age_hist": { 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "histogram": [], 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "upper_bound": 1 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "perf_stat": { 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "commit_latency_ms": 0, 2024-01-10T07:29:16.800 INFO:tasks.workunit.client.0.smithi057.stdout: "apply_latency_ms": 0, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "commit_latency_ns": 0, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "apply_latency_ns": 0 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "alerts": [], 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "network_ping_times": [] 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_sum_by_class": { 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "hdd": { 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "up_from": 0, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "seq": 0, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pgs": 4, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "num_osds": 1, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "num_per_pool_osds": 1, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "num_per_pool_omap_osds": 1, 2024-01-10T07:29:16.801 INFO:tasks.workunit.client.0.smithi057.stdout: "kb": 104857600, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used": 297212, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used_data": 116, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used_omap": 0, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_used_meta": 297088, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "kb_avail": 104560388, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "statfs": { 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "total": 107374182400, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "available": 107069837312, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "internally_reserved": 0, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "allocated": 118784, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "data_stored": 24598, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed": 0, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_allocated": 0, 2024-01-10T07:29:16.802 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_original": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "omap_allocated": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "internal_metadata": 304218112 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "hb_peers": [], 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "snap_trim_queue_len": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "num_snap_trimming": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "num_shards_repaired": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "op_queue_age_hist": { 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "histogram": [], 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "upper_bound": 1 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "perf_stat": { 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "commit_latency_ms": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "apply_latency_ms": 0, 2024-01-10T07:29:16.803 INFO:tasks.workunit.client.0.smithi057.stdout: "commit_latency_ns": 0, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "apply_latency_ns": 0 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "alerts": [], 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "network_ping_times": [] 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "pool_stats": [ 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "poolid": 1, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pg": 4, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "stat_sum": { 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "num_bytes": 19, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects": 1, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "num_object_clones": 0, 2024-01-10T07:29:16.804 INFO:tasks.workunit.client.0.smithi057.stdout: "num_object_copies": 2, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_missing_on_primary": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_missing": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_degraded": 1, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_misplaced": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_unfound": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_dirty": 1, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_whiteouts": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_read": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_read_kb": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_write": 2, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_write_kb": 2, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_scrub_errors": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_shallow_scrub_errors": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_deep_scrub_errors": 0, 2024-01-10T07:29:16.805 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_recovered": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_bytes_recovered": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_keys_recovered": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_omap": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_hit_set_archive": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_bytes_hit_set_archive": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush_kb": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict_kb": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_promote": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush_mode_high": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_flush_mode_low": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict_mode_some": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_evict_mode_full": 0, 2024-01-10T07:29:16.806 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_pinned": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "num_legacy_snapsets": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "num_large_omap_objects": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_manifest": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "num_omap_bytes": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "num_omap_keys": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "num_objects_repaired": 0 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "store_stats": { 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "total": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "available": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "internally_reserved": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "allocated": 4096, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "data_stored": 19, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed": 0, 2024-01-10T07:29:16.807 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_allocated": 0, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "data_compressed_original": 0, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "omap_allocated": 0, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "internal_metadata": 0 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "log_size": 2, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "ondisk_log_size": 2, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "up": 4, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "acting": 4, 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "num_store_stats": 1 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "osd_stats": [ 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.808 INFO:tasks.workunit.client.0.smithi057.stdout: "osd": 0, 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "seq": 21474836482 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pg_by_state": [ 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "state": "active+undersized+degraded", 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "num": 1 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "state": "active+undersized", 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "num": 3 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "num_pg_by_osd": [ 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.809 INFO:tasks.workunit.client.0.smithi057.stdout: "osd": 0, 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "num_primary_pg": 4, 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "num_acting_pg": 4, 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "num_up_not_acting_pg": 0 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "purged_snaps": [ 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: { 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "pool": 1, 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "purged_snaps": {} 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: ], 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "servicemap": { 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "epoch": 1, 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "modified": "2024-01-10T07:28:28.425392+0000", 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: "services": {} 2024-01-10T07:29:16.810 INFO:tasks.workunit.client.0.smithi057.stdout: }, 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout: "paxos": { 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout: "first_committed": 1, 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout: "last_committed": 123, 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout: "last_pn": 0, 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout: "accepted_pn": 0 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout: } 2024-01-10T07:29:16.811 INFO:tasks.workunit.client.0.smithi057.stdout:} 2024-01-10T07:29:16.815 INFO:tasks.workunit.client.0.smithi057.stderr:report 3844410464 2024-01-10T07:29:16.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-01-10T07:29:16.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-01-10T07:29:16.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:29:16.815 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:29:16.816 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:29:16.817 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e6097f03-25b6-4209-bd5b-b552307da23d 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e6097f03-25b6-4209-bd5b-b552307da23d' 2024-01-10T07:29:16.818 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 e6097f03-25b6-4209-bd5b-b552307da23d 2024-01-10T07:29:16.819 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:29:16.819 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMR55lRid9LhAAQEvyKp15M8flg999lFytyA== 2024-01-10T07:29:16.819 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMR55lRid9LhAAQEvyKp15M8flg999lFytyA=="}' 2024-01-10T07:29:16.819 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e6097f03-25b6-4209-bd5b-b552307da23d -i td/ceph-helpers/1/new.json 2024-01-10T07:29:17.354 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:29:17.367 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:29:17.368 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMR55lRid9LhAAQEvyKp15M8flg999lFytyA== --osd-uuid e6097f03-25b6-4209-bd5b-b552307da23d 2024-01-10T07:29:17.393 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.392+0000 7f1490177540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:17.394 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.394+0000 7f1490177540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:17.395 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.395+0000 7f1490177540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:17.396 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.396+0000 7f1490177540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:29:17.396 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.396+0000 7f1490177540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:29:17.396 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.396+0000 7f1490177540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:29:17.473 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:17.473+0000 7f1490177540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:29:23.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:29:23.044 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:29:23.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:29:23.045 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:29:23.045 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:29:23.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:29:23.670 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:29:23.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:29:23.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:29:23.670 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:29:23.671 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:29:23.693 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:23.693+0000 7f4cd26dd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:23.694 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:23.695+0000 7f4cd26dd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:23.695 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:23.695+0000 7f4cd26dd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:24.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:29:24.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:29:24.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:24.203 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:24.231 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:24.231+0000 7f4cd26dd540 -1 Falling back to public interface 2024-01-10T07:29:24.694 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:25.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:25.695 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:25.695 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:29:25.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:29:25.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:25.696 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:26.187 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:26.842 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:26.842+0000 7f4cd26dd540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:29:27.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:27.189 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:27.189 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:29:27.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:29:27.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:27.190 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:27.720 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:28.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:28.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:28.721 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:29:28.722 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:29:28.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:28.722 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:29.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:30.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:30.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:30.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:29:30.306 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:29:30.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:30.306 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:30.770 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:31.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:31.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:31.772 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:29:31.772 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:29:31.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:31.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:32.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:33.245 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:33.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:33.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:29:33.246 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:29:33.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:33.246 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:33.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:34.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:34.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:34.786 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:29:34.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:29:34.786 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:34.787 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:35.286 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:36.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:36.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:36.288 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:29:36.288 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:29:36.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:36.289 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:36.796 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:37.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:37.797 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:37.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:29:37.798 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:29:37.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:37.798 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:38.262 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:39.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:39.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:39.264 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:29:39.264 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:29:39.265 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:39.265 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:39.978 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:40.979 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:40.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:40.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:29:40.980 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:29:40.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:40.980 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:41.595 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:42.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:42.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:42.597 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:29:42.597 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:29:42.597 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:42.597 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:43.316 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:44.318 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:29:44.318 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:29:44.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:44.319 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:45.214 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:46.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:46.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:46.216 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:29:46.217 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:29:46.217 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:46.218 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:47.003 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:48.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:48.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:48.004 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:29:48.004 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:29:48.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:48.005 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:48.596 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:29:49.597 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:29:49.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:29:49.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:29:49.598 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:29:49.598 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:29:49.599 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:29:50.161 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/164000,v1:127.0.0.1:6811/164000] [v2:127.0.0.1:6812/164000,v1:127.0.0.1:6813/164000] exists,up e6097f03-25b6-4209-bd5b-b552307da23d 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-01-10T07:29:50.163 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:29:50.164 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:29:50.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:29:50.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:29:50.164 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:29:50.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-01-10T07:29:50.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-01-10T07:29:50.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-01-10T07:29:50.255 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2024-01-10T07:29:50.255 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2024-01-10T07:29:50.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2024-01-10T07:29:50.767 INFO:tasks.workunit.client.0.smithi057.stderr:1' 2024-01-10T07:29:50.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2024-01-10T07:29:50.767 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:29:50.767 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-01-10T07:29:50.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836492 2024-01-10T07:29:50.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836492 2024-01-10T07:29:50.946 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836492' 2024-01-10T07:29:50.947 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2024-01-10T07:29:50.947 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-01-10T07:29:51.084 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=73014444038 2024-01-10T07:29:51.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 73014444038 2024-01-10T07:29:51.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836492 1-73014444038' 2024-01-10T07:29:51.085 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:29:51.085 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836492 2024-01-10T07:29:51.085 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:29:51.086 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2024-01-10T07:29:51.086 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836492 2024-01-10T07:29:51.086 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:29:51.087 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836492 2024-01-10T07:29:51.087 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-01-10T07:29:51.087 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.0 seq 21474836492 2024-01-10T07:29:51.088 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:29:51.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836489 -lt 21474836492 2024-01-10T07:29:51.544 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2024-01-10T07:29:52.545 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2024-01-10T07:29:52.545 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2024-01-10T07:29:53.023 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836492 2024-01-10T07:29:53.024 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-73014444038 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-73014444038 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=73014444038 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 73014444038' 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stdout:waiting osd.1 seq 73014444038 2024-01-10T07:29:53.025 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2024-01-10T07:29:53.522 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 73014444038 -lt 73014444038 2024-01-10T07:29:53.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-01-10T07:29:53.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:53.523 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:54.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-01-10T07:29:54.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-01-10T07:29:54.150 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-01-10T07:29:54.150 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-01-10T07:29:54.151 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-01-10T07:29:54.151 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-01-10T07:29:54.151 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-01-10T07:29:54.151 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-01-10T07:29:54.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-01-10T07:29:54.623 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-01-10T07:29:54.623 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-01-10T07:29:54.623 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-01-10T07:29:55.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-01-10T07:29:55.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-01-10T07:29:55.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-01-10T07:29:55.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:29:55.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:29:55.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:29:55.403 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:29:55.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:29:55.404 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:29:55.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:29:55.405 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:29:55.405 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:29:55.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:29:55.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:29:55.407 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:29:55.407 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:29:55.408 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:29:55.409 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:29:55.409 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:29:55.418 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:29:55.418 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:55.418 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:55.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:29:55.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: test_wait_for_health_ok: setup td/ceph-helpers 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:29:55.421 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:29:55.422 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:29:55.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:29:55.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:29:55.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:29:55.422 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:29:55.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:29:55.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:29:55.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:29:55.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:29:55.423 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:29:55.423 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:29:55.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:29:55.425 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:29:55.426 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:29:55.426 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:29:55.426 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:29:55.427 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-01-10T07:29:55.427 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:29:55.428 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:29:55.429 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:55.429 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:55.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:29:55.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:29:55.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:29:55.429 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-01-10T07:29:55.431 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-01-10T07:29:55.431 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:55.431 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:55.431 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87895 2024-01-10T07:29:55.433 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-01-10T07:29:55.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-01-10T07:29:55.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-01-10T07:29:55.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-01-10T07:29:55.433 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-01-10T07:29:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-01-10T07:29:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-01-10T07:29:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-01-10T07:29:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-01-10T07:29:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-01-10T07:29:55.434 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-01-10T07:29:55.846 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-01-10T07:29:55.846 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:29:55.846 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:29:55.846 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:29:55.847 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:55.847 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:55.847 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:29:55.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-01-10T07:29:56.097 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-01-10T07:29:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-01-10T07:29:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:29:56.098 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:29:56.099 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-01-10T07:29:56.099 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:29:56.100 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get fsid 2024-01-10T07:29:56.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-01-10T07:29:56.206 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87895/ceph-mon.a.asok 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-01-10T07:29:56.207 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87895/ceph-mon.a.asok config get mon_host 2024-01-10T07:29:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1787: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-01-10T07:29:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-01-10T07:29:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-01-10T07:29:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-01-10T07:29:56.307 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-01-10T07:29:56.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-01-10T07:29:56.308 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-01-10T07:29:57.209 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-01-10T07:29:57.209 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-01-10T07:29:57.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:29:57.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:29:57.242 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:29:57.243 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:29:57.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0d15bbf-26f3-4c19-8522-aa868de95132 2024-01-10T07:29:57.244 INFO:tasks.workunit.client.0.smithi057.stdout:add osd0 b0d15bbf-26f3-4c19-8522-aa868de95132 2024-01-10T07:29:57.244 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b0d15bbf-26f3-4c19-8522-aa868de95132' 2024-01-10T07:29:57.244 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:29:57.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1R55l0lNeDxAAGDoN1mpxotitdBR3kJl2pA== 2024-01-10T07:29:57.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1R55l0lNeDxAAGDoN1mpxotitdBR3kJl2pA=="}' 2024-01-10T07:29:57.258 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0d15bbf-26f3-4c19-8522-aa868de95132 -i td/ceph-helpers/0/new.json 2024-01-10T07:29:57.656 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:29:57.668 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-01-10T07:29:57.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1R55l0lNeDxAAGDoN1mpxotitdBR3kJl2pA== --osd-uuid b0d15bbf-26f3-4c19-8522-aa868de95132 2024-01-10T07:29:57.694 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.694+0000 7f4cda4b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:57.696 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.696+0000 7f4cda4b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:57.697 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.697+0000 7f4cda4b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:29:57.697 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.697+0000 7f4cda4b0540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:29:57.698 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.697+0000 7f4cda4b0540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:29:57.698 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.697+0000 7f4cda4b0540 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-01-10T07:29:57.768 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:29:57.768+0000 7f4cda4b0540 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-01-10T07:30:03.360 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-01-10T07:30:03.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:30:03.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-01-10T07:30:03.362 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd0 key to auth repository 2024-01-10T07:30:03.362 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:30:03.611 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:30:03.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-01-10T07:30:03.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:30:03.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:30:03.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:30:03.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:30:03.636 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:03.636+0000 7feb3390e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:03.650 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:03.650+0000 7feb3390e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:03.651 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:03.651+0000 7feb3390e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:04.107 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-01-10T07:30:04.107 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:30:04.108 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:30:04.108 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:30:04.108 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:30:04.108 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:04.108 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:30:04.108 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:30:04.109 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:04.109 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:04.197 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:04.197+0000 7feb3390e540 -1 Falling back to public interface 2024-01-10T07:30:04.607 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:05.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:05.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:05.609 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:30:05.609 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:30:05.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:05.610 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:06.092 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:06.722 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:06.722+0000 7feb3390e540 -1 osd.0 0 log_to_monitors true 2024-01-10T07:30:07.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:07.094 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:07.094 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:30:07.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:30:07.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:07.095 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:07.761 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:08.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:08.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:08.763 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:30:08.763 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:30:08.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:08.764 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:09.228 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:10.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:10.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:10.229 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:30:10.230 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:30:10.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:10.230 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:10.741 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:11.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:11.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:11.743 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:30:11.743 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:30:11.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:11.744 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:12.200 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:13.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:13.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:13.201 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:30:13.202 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:30:13.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:13.202 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:13.771 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:14.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:14.773 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:14.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:30:14.774 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:30:14.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:14.774 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:15.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:16.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:16.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:16.240 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:30:16.240 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:30:16.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:16.241 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:16.849 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:17.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:17.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:17.852 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:30:17.852 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:30:17.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:17.853 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:18.782 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:19.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:19.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:19.784 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:30:19.784 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:30:19.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:19.785 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:20.727 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:21.728 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:21.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:21.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:30:21.729 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:30:21.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:21.729 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:22.369 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:23.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:23.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:23.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:30:23.370 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:30:23.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:23.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:24.270 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:25.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:25.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:25.272 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:30:25.272 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:30:25.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:25.273 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:26.303 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:27.304 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:27.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:27.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:30:27.305 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:30:27.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:27.305 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:28.103 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:29.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:29.104 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:29.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:30:29.105 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:30:29.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:29.105 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:30:29.568 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/166407,v1:127.0.0.1:6803/166407] [v2:127.0.0.1:6804/166407,v1:127.0.0.1:6805/166407] exists,up b0d15bbf-26f3-4c19-8522-aa868de95132 2024-01-10T07:30:29.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:30:29.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:30:29.572 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:30:29.573 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:30:29.574 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d74d8c2-6180-488d-9f49-ee7c37967843 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6d74d8c2-6180-488d-9f49-ee7c37967843' 2024-01-10T07:30:29.575 INFO:tasks.workunit.client.0.smithi057.stdout:add osd1 6d74d8c2-6180-488d-9f49-ee7c37967843 2024-01-10T07:30:29.576 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:30:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVR55lFl4LIxAANkrThAy7E7irfepgHGG1Rw== 2024-01-10T07:30:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVR55lFl4LIxAANkrThAy7E7irfepgHGG1Rw=="}' 2024-01-10T07:30:29.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d74d8c2-6180-488d-9f49-ee7c37967843 -i td/ceph-helpers/1/new.json 2024-01-10T07:30:30.166 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:30:30.179 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-01-10T07:30:30.180 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVR55lFl4LIxAANkrThAy7E7irfepgHGG1Rw== --osd-uuid 6d74d8c2-6180-488d-9f49-ee7c37967843 2024-01-10T07:30:30.204 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.204+0000 7fc67126d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:30.206 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.206+0000 7fc67126d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:30.207 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.207+0000 7fc67126d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:30.208 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.207+0000 7fc67126d540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:30:30.208 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.207+0000 7fc67126d540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:30:30.208 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.207+0000 7fc67126d540 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-01-10T07:30:30.380 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:30.380+0000 7fc67126d540 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-01-10T07:30:36.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-01-10T07:30:36.276 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:30:36.277 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd1 key to auth repository 2024-01-10T07:30:36.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-01-10T07:30:36.278 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:30:36.905 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-01-10T07:30:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:30:36.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:30:36.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:30:36.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:30:36.906 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:30:36.928 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:36.928+0000 7f8e9fb9f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:36.930 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:36.930+0000 7f8e9fb9f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:36.931 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:36.931+0000 7f8e9fb9f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:30:37.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-01-10T07:30:37.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:30:37.401 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:30:37.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:30:37.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:30:37.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:37.402 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:30:37.402 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:30:37.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:37.403 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:37.449 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:37.449+0000 7f8e9fb9f540 -1 Falling back to public interface 2024-01-10T07:30:37.885 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:38.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:38.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:38.886 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:30:38.887 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:30:38.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:38.887 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:39.370 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:39.976 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:30:39.976+0000 7f8e9fb9f540 -1 osd.1 0 log_to_monitors true 2024-01-10T07:30:40.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:40.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:40.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:30:40.372 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:30:40.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:40.372 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:40.933 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:41.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:41.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:41.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:30:41.935 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:30:41.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:41.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:42.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:43.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:43.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:43.396 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:30:43.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:30:43.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:43.397 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:43.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:44.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:44.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:44.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:30:44.928 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:30:44.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:44.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:45.418 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:46.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:46.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:46.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:30:46.420 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:30:46.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:46.421 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:46.925 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:47.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:47.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:47.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:30:47.927 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:30:47.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:47.928 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:48.393 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:49.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:49.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:49.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:30:49.395 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:30:49.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:49.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:49.890 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:50.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:50.891 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:50.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:30:50.892 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:30:50.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:50.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:51.353 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:52.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:52.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:52.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:30:52.354 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:30:52.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:52.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:52.823 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:53.824 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:53.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:53.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:30:53.825 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:30:53.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:53.825 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:54.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:55.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:55.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:55.479 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:30:55.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:30:55.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:55.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:56.059 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:57.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:57.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:57.061 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:30:57.061 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:30:57.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:57.062 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:57.848 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:30:58.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:30:58.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:30:58.846 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:30:58.847 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:30:58.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:30:58.847 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:30:59.890 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:00.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:00.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:00.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:31:00.892 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:31:00.892 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:00.893 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:31:01.503 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:02.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:02.505 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:02.505 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:31:02.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:31:02.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:02.506 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:31:02.981 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/168134,v1:127.0.0.1:6811/168134] [v2:127.0.0.1:6812/168134,v1:127.0.0.1:6813/168134] exists,up 6d74d8c2-6180-488d-9f49-ee7c37967843 2024-01-10T07:31:02.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:31:02.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:31:02.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:31:02.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:31:02.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-01-10T07:31:02.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:31:02.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:02.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:02.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:31:02.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-01-10T07:31:02.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-01-10T07:31:02.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:31:02.993 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-01-10T07:31:02.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-01-10T07:31:02.995 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-01-10T07:31:02.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce48113b-e597-40c1-b657-98437e5adee7 2024-01-10T07:31:02.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ce48113b-e597-40c1-b657-98437e5adee7' 2024-01-10T07:31:02.995 INFO:tasks.workunit.client.0.smithi057.stdout:add osd2 ce48113b-e597-40c1-b657-98437e5adee7 2024-01-10T07:31:02.995 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-01-10T07:31:03.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3R55lZPwDABAA4y+NelzkjOyyFFMCTfBzpQ== 2024-01-10T07:31:03.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3R55lZPwDABAA4y+NelzkjOyyFFMCTfBzpQ=="}' 2024-01-10T07:31:03.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce48113b-e597-40c1-b657-98437e5adee7 -i td/ceph-helpers/2/new.json 2024-01-10T07:31:03.625 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:31:03.637 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-01-10T07:31:03.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3R55lZPwDABAA4y+NelzkjOyyFFMCTfBzpQ== --osd-uuid ce48113b-e597-40c1-b657-98437e5adee7 2024-01-10T07:31:03.663 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.663+0000 7fcf706ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:03.665 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.665+0000 7fcf706ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:03.666 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.666+0000 7fcf706ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:03.666 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.666+0000 7fcf706ac540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:31:03.666 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.666+0000 7fcf706ac540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:31:03.666 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.666+0000 7fcf706ac540 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-01-10T07:31:03.711 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:03.711+0000 7fcf706ac540 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-01-10T07:31:09.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-01-10T07:31:09.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-01-10T07:31:09.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-01-10T07:31:09.995 INFO:tasks.workunit.client.0.smithi057.stdout:adding osd2 key to auth repository 2024-01-10T07:31:09.995 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-01-10T07:31:10.627 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T07:31:10.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-01-10T07:31:10.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:31:10.628 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-01-10T07:31:10.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-01-10T07:31:10.629 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-01-10T07:31:10.651 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:10.651+0000 7f73ba784540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:10.653 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:10.653+0000 7f73ba784540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:10.654 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:10.654+0000 7f73ba784540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:11.120 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:31:11.121 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:31:11.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:11.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:11.171 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:11.171+0000 7f73ba784540 -1 Falling back to public interface 2024-01-10T07:31:11.621 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:12.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:12.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:12.622 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:31:12.622 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:31:12.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:12.623 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:13.119 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:13.729 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:13.729+0000 7f73ba784540 -1 osd.2 0 log_to_monitors true 2024-01-10T07:31:14.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:14.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:14.121 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:31:14.121 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:31:14.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:14.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:14.699 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:15.700 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:15.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:15.703 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-01-10T07:31:15.704 INFO:tasks.workunit.client.0.smithi057.stdout:3 2024-01-10T07:31:15.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:15.704 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:16.159 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:17.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:17.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:17.160 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-01-10T07:31:17.160 INFO:tasks.workunit.client.0.smithi057.stdout:4 2024-01-10T07:31:17.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:17.161 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:17.659 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:18.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:18.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:18.660 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-01-10T07:31:18.660 INFO:tasks.workunit.client.0.smithi057.stdout:5 2024-01-10T07:31:18.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:18.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:19.122 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:20.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:20.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:20.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-01-10T07:31:20.124 INFO:tasks.workunit.client.0.smithi057.stdout:6 2024-01-10T07:31:20.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:20.124 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:20.661 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:21.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:21.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:21.662 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-01-10T07:31:21.662 INFO:tasks.workunit.client.0.smithi057.stdout:7 2024-01-10T07:31:21.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:21.663 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:22.149 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:23.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:23.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:23.150 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-01-10T07:31:23.151 INFO:tasks.workunit.client.0.smithi057.stdout:8 2024-01-10T07:31:23.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:23.151 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:23.669 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:24.671 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:24.671 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:24.671 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-01-10T07:31:24.671 INFO:tasks.workunit.client.0.smithi057.stdout:9 2024-01-10T07:31:24.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:24.672 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:25.132 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:26.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:26.133 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:26.134 INFO:tasks.workunit.client.0.smithi057.stdout:10 2024-01-10T07:31:26.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-01-10T07:31:26.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:26.134 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:26.970 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:27.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:27.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:27.971 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-01-10T07:31:27.972 INFO:tasks.workunit.client.0.smithi057.stdout:11 2024-01-10T07:31:27.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:27.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:28.638 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:29.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:29.639 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:29.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-01-10T07:31:29.640 INFO:tasks.workunit.client.0.smithi057.stdout:12 2024-01-10T07:31:29.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:29.640 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:30.583 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:31.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:31.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:31.585 INFO:tasks.workunit.client.0.smithi057.stdout:13 2024-01-10T07:31:31.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-01-10T07:31:31.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:31.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:32.352 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:33.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:33.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:33.354 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-01-10T07:31:33.354 INFO:tasks.workunit.client.0.smithi057.stdout:14 2024-01-10T07:31:33.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:33.355 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:34.336 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:35.337 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:35.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:35.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-01-10T07:31:35.338 INFO:tasks.workunit.client.0.smithi057.stdout:15 2024-01-10T07:31:35.338 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:35.339 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:35.929 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:36.930 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:36.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:36.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-01-10T07:31:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:16 2024-01-10T07:31:36.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:36.931 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:37.385 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/169891,v1:127.0.0.1:6819/169891] [v2:127.0.0.1:6820/169891,v1:127.0.0.1:6821/169891] exists,up ce48113b-e597-40c1-b657-98437e5adee7 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:31:37.387 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:31:37.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:31:37.693 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: test_wait_for_health_ok: ceph osd down 0 2024-01-10T07:31:38.199 INFO:tasks.workunit.client.0.smithi057.stderr:osd.0 is already down. 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: test_wait_for_health_ok: TIMEOUT=1 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: test_wait_for_health_ok: wait_for_health_ok 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: wait_for_health HEALTH_OK 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: wait_for_health: local grepstr=HEALTH_OK 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: delays=($(get_timeout_delays $TIMEOUT .1)) 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: get_timeout_delays 1 .1 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:31:38.210 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:31:38.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: local -a delays 2024-01-10T07:31:38.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: wait_for_health: local -i loop=0 2024-01-10T07:31:38.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:38.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:38.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 0 >= 4 )) 2024-01-10T07:31:38.871 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.1 2024-01-10T07:31:38.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:38.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:38.972 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:39.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 1 >= 4 )) 2024-01-10T07:31:39.590 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.2 2024-01-10T07:31:39.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:39.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:39.791 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:40.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 2 >= 4 )) 2024-01-10T07:31:40.420 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.4 2024-01-10T07:31:40.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:40.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:40.821 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:41.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 3 >= 4 )) 2024-01-10T07:31:41.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.3 2024-01-10T07:31:41.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:41.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:41.742 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:42.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 4 >= 4 )) 2024-01-10T07:31:42.373 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1764: wait_for_health: ceph health detail 2024-01-10T07:31:42.969 INFO:tasks.workunit.client.0.smithi057.stdout:HEALTH_ERR 2 mgr modules have failed; 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout:[ERR] MGR_MODULE_ERROR: 2 mgr modules have failed 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: Module 'rbd_support' has failed: Not found or unloadable 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: Module 'volumes' has failed: Not found or unloadable 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout:[WRN] OSD_DOWN: 3 osds down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: osd.0 (root=default,host=smithi057) is down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: osd.1 (root=default,host=smithi057) is down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: osd.2 (root=default,host=smithi057) is down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: host smithi057 (root=default) (3 osds) is down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-01-10T07:31:42.970 INFO:tasks.workunit.client.0.smithi057.stdout: root default (3 osds) is down 2024-01-10T07:31:42.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1765: wait_for_health: return 1 2024-01-10T07:31:42.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: return 1 2024-01-10T07:31:42.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-01-10T07:31:42.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:31:42.986 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:31:42.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:31:42.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:31:42.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-01-10T07:31:42.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-01-10T07:31:42.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-01-10T07:31:42.988 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.0 2024-01-10T07:31:42.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:31:42.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-01-10T07:31:42.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-01-10T07:31:42.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:31:42.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:31:42.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:31:43.007 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:43.007+0000 7f279160f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:43.009 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:43.009+0000 7f279160f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:43.010 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:43.010+0000 7f279160f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:43.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-01-10T07:31:43.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:31:43.479 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:43.480 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:31:43.529 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:43.529+0000 7f279160f540 -1 Falling back to public interface 2024-01-10T07:31:43.954 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:44.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:44.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:44.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:31:44.956 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:31:44.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:44.956 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:31:45.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:46.439 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:46.439+0000 7f279160f540 -1 osd.0 14 log_to_monitors true 2024-01-10T07:31:46.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:46.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:46.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:31:46.440 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:31:46.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:46.440 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-01-10T07:31:46.986 INFO:tasks.workunit.client.0.smithi057.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,14) [v2:127.0.0.1:6802/171043,v1:127.0.0.1:6803/171043] [v2:127.0.0.1:6804/171043,v1:127.0.0.1:6805/171043] exists,up b0d15bbf-26f3-4c19-8522-aa868de95132 2024-01-10T07:31:46.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:31:46.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:31:46.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:31:46.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:31:46.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:31:46.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:31:46.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-01-10T07:31:46.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-01-10T07:31:46.992 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.1 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:31:46.994 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:31:47.011 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:47.011+0000 7fee4827b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:47.013 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:47.013+0000 7fee4827b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:47.014 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:47.014+0000 7fee4827b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:47.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:31:47.459 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:31:47.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:47.459 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:31:47.534 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:47.534+0000 7fee4827b540 -1 Falling back to public interface 2024-01-10T07:31:47.934 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:48.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:48.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:48.935 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:31:48.935 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:31:48.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:48.936 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:31:49.436 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:50.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:50.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:50.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:31:50.437 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:31:50.437 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:50.438 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-01-10T07:31:50.444 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:50.444+0000 7fee4827b540 -1 osd.1 16 log_to_monitors true 2024-01-10T07:31:50.984 INFO:tasks.workunit.client.0.smithi057.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/171600,v1:127.0.0.1:6811/171600] [v2:127.0.0.1:6812/171600,v1:127.0.0.1:6813/171600] exists,up 6d74d8c2-6180-488d-9f49-ee7c37967843 2024-01-10T07:31:50.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:31:50.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:31:50.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:31:50.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-01-10T07:31:50.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-01-10T07:31:50.988 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-01-10T07:31:50.989 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-01-10T07:31:50.990 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-01-10T07:31:50.991 INFO:tasks.workunit.client.0.smithi057.stdout:start osd.2 2024-01-10T07:31:50.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=c182505b-1213-4a65-a0b1-c14e269ca5fa --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.87895/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-01-10T07:31:50.992 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-01-10T07:31:50.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-01-10T07:31:50.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-01-10T07:31:50.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-01-10T07:31:50.992 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-01-10T07:31:51.009 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:51.009+0000 7fc1d19f9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:51.040 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:51.040+0000 7fc1d19f9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:51.041 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:51.041+0000 7fc1d19f9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-01-10T07:31:51.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-01-10T07:31:51.471 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-01-10T07:31:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-01-10T07:31:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-01-10T07:31:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-01-10T07:31:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-01-10T07:31:51.472 INFO:tasks.workunit.client.0.smithi057.stdout:0 2024-01-10T07:31:51.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:51.473 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:51.571 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:51.571+0000 7fc1d19f9540 -1 Falling back to public interface 2024-01-10T07:31:51.955 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:52.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:52.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:52.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-01-10T07:31:52.957 INFO:tasks.workunit.client.0.smithi057.stdout:1 2024-01-10T07:31:52.957 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:52.958 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:53.441 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-01-10T07:31:54.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-01-10T07:31:54.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-01-10T07:31:54.443 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-01-10T07:31:54.443 INFO:tasks.workunit.client.0.smithi057.stdout:2 2024-01-10T07:31:54.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-01-10T07:31:54.444 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-01-10T07:31:54.623 INFO:tasks.workunit.client.0.smithi057.stderr:2024-01-10T07:31:54.622+0000 7fc1d19f9540 -1 osd.2 15 log_to_monitors true 2024-01-10T07:31:54.997 INFO:tasks.workunit.client.0.smithi057.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6818/172157,v1:127.0.0.1:6819/172157] [v2:127.0.0.1:6820/172157,v1:127.0.0.1:6821/172157] exists,up ce48113b-e597-40c1-b657-98437e5adee7 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: test_wait_for_health_ok: wait_for_health_ok 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: wait_for_health HEALTH_OK 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: wait_for_health: local grepstr=HEALTH_OK 2024-01-10T07:31:54.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: delays=($(get_timeout_delays $TIMEOUT .1)) 2024-01-10T07:31:55.000 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: get_timeout_delays 300 .1 2024-01-10T07:31:55.000 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-01-10T07:31:55.000 INFO:tasks.workunit.client.0.smithi057.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-01-10T07:31:55.000 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-01-10T07:31:55.000 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-01-10T07:31:55.000 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-01-10T07:31:55.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: local -a delays 2024-01-10T07:31:55.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: wait_for_health: local -i loop=0 2024-01-10T07:31:55.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:55.193 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:55.799 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 0 >= 27 )) 2024-01-10T07:31:55.800 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.1 2024-01-10T07:31:55.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:55.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:55.901 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:56.517 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 1 >= 27 )) 2024-01-10T07:31:56.517 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.2 2024-01-10T07:31:56.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:56.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:56.718 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:57.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 2 >= 27 )) 2024-01-10T07:31:57.331 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.4 2024-01-10T07:31:57.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:57.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:57.732 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:58.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 3 >= 27 )) 2024-01-10T07:31:58.371 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.8 2024-01-10T07:31:59.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:31:59.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:31:59.172 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:31:59.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 4 >= 27 )) 2024-01-10T07:31:59.794 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 1.6 2024-01-10T07:32:01.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:32:01.395 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:32:01.396 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:32:02.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 5 >= 27 )) 2024-01-10T07:32:02.019 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 3.2 2024-01-10T07:32:05.220 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:32:05.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:32:05.221 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:32:05.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 6 >= 27 )) 2024-01-10T07:32:05.850 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 6.4 2024-01-10T07:32:12.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:32:12.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:32:12.251 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:32:12.923 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 7 >= 27 )) 2024-01-10T07:32:12.924 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 12.8 2024-01-10T07:32:25.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:32:25.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:32:25.725 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:32:26.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 8 >= 27 )) 2024-01-10T07:32:26.361 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:32:41.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:32:41.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:32:41.363 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:32:41.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 9 >= 27 )) 2024-01-10T07:32:41.985 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:32:56.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:32:56.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:32:56.987 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:32:57.611 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 10 >= 27 )) 2024-01-10T07:32:57.612 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:33:12.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:33:12.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:33:12.613 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:33:13.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 11 >= 27 )) 2024-01-10T07:33:13.236 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:33:28.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:33:28.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:33:28.238 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:33:28.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 12 >= 27 )) 2024-01-10T07:33:28.864 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:33:43.865 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:33:43.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:33:43.866 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:33:44.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 13 >= 27 )) 2024-01-10T07:33:44.487 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:33:59.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:33:59.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:33:59.489 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:34:00.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 14 >= 27 )) 2024-01-10T07:34:00.123 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:34:15.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:34:15.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:34:15.125 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:34:15.757 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 15 >= 27 )) 2024-01-10T07:34:15.758 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:34:30.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:34:30.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:34:30.759 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:34:31.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 16 >= 27 )) 2024-01-10T07:34:31.380 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:34:46.381 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:34:46.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:34:46.382 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:34:46.999 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 17 >= 27 )) 2024-01-10T07:34:47.000 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:35:02.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:35:02.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:35:02.001 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:35:02.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 18 >= 27 )) 2024-01-10T07:35:02.665 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:35:17.666 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:35:17.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:35:17.667 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:35:18.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 19 >= 27 )) 2024-01-10T07:35:18.295 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:35:33.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:35:33.296 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:35:33.297 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:35:33.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 20 >= 27 )) 2024-01-10T07:35:33.927 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:35:48.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:35:48.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:35:48.926 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:35:49.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 21 >= 27 )) 2024-01-10T07:35:49.557 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:36:04.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:36:04.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:36:04.559 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:36:05.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 22 >= 27 )) 2024-01-10T07:36:05.181 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:36:20.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:36:20.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:36:20.183 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:36:20.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 23 >= 27 )) 2024-01-10T07:36:20.805 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:36:35.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:36:35.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:36:35.808 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:36:36.456 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 24 >= 27 )) 2024-01-10T07:36:36.457 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:36:51.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:36:51.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:36:51.458 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:36:52.081 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 25 >= 27 )) 2024-01-10T07:36:52.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2024-01-10T07:37:07.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:37:07.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:37:07.082 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:37:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 26 >= 27 )) 2024-01-10T07:37:07.705 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 4.5 2024-01-10T07:37:12.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2024-01-10T07:37:12.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2024-01-10T07:37:12.207 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2024-01-10T07:37:12.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 27 >= 27 )) 2024-01-10T07:37:12.828 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1764: wait_for_health: ceph health detail 2024-01-10T07:37:13.463 INFO:tasks.workunit.client.0.smithi057.stdout:HEALTH_ERR 2 mgr modules have failed 2024-01-10T07:37:13.463 INFO:tasks.workunit.client.0.smithi057.stdout:[ERR] MGR_MODULE_ERROR: 2 mgr modules have failed 2024-01-10T07:37:13.464 INFO:tasks.workunit.client.0.smithi057.stdout: Module 'rbd_support' has failed: Not found or unloadable 2024-01-10T07:37:13.464 INFO:tasks.workunit.client.0.smithi057.stdout: Module 'volumes' has failed: Not found or unloadable 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1765: wait_for_health: return 1 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: return 1 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: test_wait_for_health_ok: return 1 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: run_tests: teardown td/ceph-helpers 1 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-01-10T07:37:13.477 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-01-10T07:37:13.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-01-10T07:37:13.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-01-10T07:37:13.478 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-01-10T07:37:13.584 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-01-10T07:37:13.584 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-01-10T07:37:13.585 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-01-10T07:37:13.585 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-01-10T07:37:13.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-01-10T07:37:13.586 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-01-10T07:37:13.587 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-01-10T07:37:13.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:37:13.588 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-01-10T07:37:13.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-01-10T07:37:13.589 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-10T07:37:13.589 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:37:13.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-01-10T07:37:13.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-01-10T07:37:13.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-01-10T07:37:13.591 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log td/ceph-helpers/osd.1.log td/ceph-helpers/osd.2.log /home/ubuntu/cephtest/archive/log 2024-01-10T07:37:13.592 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-01-10T07:37:13.600 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-01-10T07:37:13.600 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-01-10T07:37:13.601 INFO:tasks.workunit.client.0.smithi057.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87895 2024-01-10T07:37:13.601 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87895 2024-01-10T07:37:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T07:37:13.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-01-10T07:37:13.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-01-10T07:37:13.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: run_tests: return 1 2024-01-10T07:37:13.603 INFO:tasks.workunit.client.0.smithi057.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: main: exit 1 2024-01-10T07:37:13.604 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-01-10T07:37:13.604 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-10T07:37:13.897 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-01-10T07:37:14.115 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ff353c5bf5234cec8770b87db091caa8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_affad57d5ae9c6b2f7579f7a454052bf9428828f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-01-10T07:37:14.117 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-10T07:37:14.126 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-10T07:37:14.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-10T07:37:14.180 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-10T07:37:14.188 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-10T07:37:14.189 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-10T07:37:14.216 INFO:teuthology.orchestra.run.smithi057.stderr:bash: ntpq: command not found 2024-01-10T07:37:14.220 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-10T07:37:14.221 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-01-10T07:37:14.221 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +565us[ +615us] +/- 67ms 2024-01-10T07:37:14.221 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 +255us[ +304us] +/- 69ms 2024-01-10T07:37:14.221 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 18 -736us[ -736us] +/- 74ms 2024-01-10T07:37:14.221 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-10T07:37:14.222 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-10T07:37:14.232 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-10T07:37:14.232 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-10T07:37:14.241 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-10T07:37:14.330 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-10T07:37:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T07:37:14.361 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-10T07:37:14.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-10T07:37:14.381 INFO:teuthology.task.internal:Duration was 3556.986895 seconds 2024-01-10T07:37:14.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-10T07:37:14.393 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-10T07:37:14.393 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-10T07:37:14.448 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-10T07:37:14.888 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-10T07:37:14.888 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2024-01-10T07:37:14.888 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-10T07:37:14.913 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-10T07:37:14.913 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-10T07:37:14.996 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-10T07:37:14.997 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-10T07:37:15.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-10T07:37:15.117 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-10T07:37:15.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-10T07:37:15.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-10T07:37:15.155 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:37:15.212 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2024-01-10T07:37:15.234 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-10T07:37:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T07:37:15.292 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-10T07:37:15.302 INFO:teuthology.task.internal:Transferring archived files... 2024-01-10T07:37:15.304 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi/7510458/remote/smithi057 2024-01-10T07:37:15.305 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-10T07:37:16.483 INFO:teuthology.task.internal:Removing archive directory... 2024-01-10T07:37:16.484 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-10T07:37:16.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-10T07:37:16.545 INFO:teuthology.task.internal:Not uploading archives. 2024-01-10T07:37:16.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-10T07:37:16.563 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-10T07:37:16.563 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-10T07:37:16.597 INFO:teuthology.orchestra.run.smithi057.stdout: 526083 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 10 07:37 /home/ubuntu/cephtest 2024-01-10T07:37:16.597 INFO:teuthology.orchestra.run.smithi057.stdout: 789911 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 10 06:47 /home/ubuntu/cephtest/mnt.0 2024-01-10T07:37:16.598 INFO:teuthology.orchestra.run.smithi057.stdout: 789912 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 10 07:08 /home/ubuntu/cephtest/mnt.0/client.0 2024-01-10T07:37:16.598 INFO:teuthology.orchestra.run.smithi057.stdout: 1311282 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 10 07:08 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-10T07:37:16.598 INFO:teuthology.orchestra.run.smithi057.stdout: 1311283 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 10 07:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-01-10T07:37:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-10T07:37:16.599 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-01-10T07:37:16.599 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-01-10T07:37:16.604 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-10T07:37:16.615 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-10T07:37:16.625 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi057 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-01-10T07:37:16.625 INFO:teuthology.nuke:Checking targets against current locks 2024-01-10T07:37:16.654 DEBUG:teuthology.nuke:shortname: smithi057 2024-01-10T07:37:16.654 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-10T07:37:16.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-09_20:55:51-rados-quincy-distro-default-smithi/7510458', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-10 06:28:57.938045', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMaxpaq1R0hS1TYBzZx6wJyUceMDMiSVe4L408tP2oyOEUIeTZmBCcO+KxNQfFBjWiw7K/PM07hw/izD6tbKyUc='} 2024-01-10T07:37:16.734 INFO:teuthology.orchestra.console.smithi057:Power off 2024-01-10T07:37:16.734 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-10T07:37:16.757 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-01-10T07:37:16.758 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-10T07:37:16.783 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-01-10T07:37:20.785 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-10T07:37:20.895 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-01-10T07:37:24.900 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-10T07:37:25.010 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-01-10T07:37:25.111 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-01-10T07:37:25.217 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_8} workloads/misc} duration: 3556.986895084381 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on smithi057 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=affad57d5ae9c6b2f7579f7a454052bf9428828f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ff353c5bf5234cec8770b87db091caa8 status: fail success: false 2024-01-10T07:37:25.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-10T07:37:25.305 INFO:teuthology.run:FAIL